2023-04-01 14:53:32,656 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71 2023-04-01 14:53:32,672 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-01 14:53:32,688 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-01 14:53:32,689 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/cluster_9b9f408e-8fe1-e5f9-3143-13e52c404ab9, deleteOnExit=true 2023-04-01 14:53:32,689 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-01 14:53:32,690 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/test.cache.data in system properties and HBase conf 2023-04-01 14:53:32,690 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/hadoop.tmp.dir in system properties and HBase conf 2023-04-01 14:53:32,691 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/hadoop.log.dir in system properties and HBase conf 2023-04-01 14:53:32,692 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-01 14:53:32,692 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-01 14:53:32,692 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-01 14:53:32,820 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-01 14:53:33,243 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-01 14:53:33,248 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-01 14:53:33,249 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-01 14:53:33,249 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-01 14:53:33,249 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-01 14:53:33,250 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-01 14:53:33,250 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-01 14:53:33,250 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-01 14:53:33,250 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-01 14:53:33,251 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-01 14:53:33,251 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/nfs.dump.dir in system properties and HBase conf 2023-04-01 14:53:33,252 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/java.io.tmpdir in system properties and HBase conf 2023-04-01 14:53:33,252 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-01 14:53:33,253 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-01 14:53:33,253 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-01 14:53:33,850 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-01 14:53:33,854 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-01 14:53:34,155 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-01 14:53:34,347 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-01 14:53:34,363 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-01 14:53:34,402 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-01 14:53:34,436 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/java.io.tmpdir/Jetty_localhost_44163_hdfs____.erpico/webapp 2023-04-01 14:53:34,577 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44163 2023-04-01 14:53:34,614 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-01 14:53:34,614 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-01 14:53:35,059 WARN [Listener at localhost/39359] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-01 14:53:35,206 WARN [Listener at localhost/39359] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-01 14:53:35,224 WARN [Listener at localhost/39359] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-01 14:53:35,231 INFO [Listener at localhost/39359] log.Slf4jLog(67): jetty-6.1.26 2023-04-01 14:53:35,237 INFO [Listener at localhost/39359] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/java.io.tmpdir/Jetty_localhost_42345_datanode____dcde3h/webapp 2023-04-01 14:53:35,339 INFO [Listener at localhost/39359] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42345 2023-04-01 14:53:35,773 WARN [Listener at localhost/46667] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-01 14:53:36,324 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x490f0df6ccf1ed9f: Processing first storage report for DS-4d3588f6-7a66-4dcd-af22-6baef11c9ddb from datanode 350c837c-6d4c-42a4-a453-bc0081ccae8d 2023-04-01 14:53:36,326 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x490f0df6ccf1ed9f: from storage DS-4d3588f6-7a66-4dcd-af22-6baef11c9ddb node DatanodeRegistration(127.0.0.1:43281, datanodeUuid=350c837c-6d4c-42a4-a453-bc0081ccae8d, infoPort=32967, infoSecurePort=0, ipcPort=46667, storageInfo=lv=-57;cid=testClusterID;nsid=1691005520;c=1680360813934), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-01 14:53:36,326 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x490f0df6ccf1ed9f: Processing first storage report for DS-575cbb77-62e5-495c-8018-22e41807f21a from datanode 350c837c-6d4c-42a4-a453-bc0081ccae8d 2023-04-01 14:53:36,326 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x490f0df6ccf1ed9f: from storage DS-575cbb77-62e5-495c-8018-22e41807f21a node DatanodeRegistration(127.0.0.1:43281, datanodeUuid=350c837c-6d4c-42a4-a453-bc0081ccae8d, infoPort=32967, infoSecurePort=0, ipcPort=46667, storageInfo=lv=-57;cid=testClusterID;nsid=1691005520;c=1680360813934), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-01 14:53:36,372 DEBUG [Listener at localhost/46667] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71 2023-04-01 14:53:36,436 INFO [Listener at localhost/46667] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/cluster_9b9f408e-8fe1-e5f9-3143-13e52c404ab9/zookeeper_0, clientPort=60465, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/cluster_9b9f408e-8fe1-e5f9-3143-13e52c404ab9/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/cluster_9b9f408e-8fe1-e5f9-3143-13e52c404ab9/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-01 14:53:36,449 INFO [Listener at localhost/46667] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60465 2023-04-01 14:53:36,457 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 14:53:36,459 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 14:53:37,151 INFO [Listener at localhost/46667] util.FSUtils(471): Created version file at hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a with version=8 2023-04-01 14:53:37,151 INFO [Listener at localhost/46667] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/hbase-staging 2023-04-01 14:53:37,466 INFO [Listener at localhost/46667] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-01 14:53:37,981 INFO [Listener at localhost/46667] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-01 14:53:38,017 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-01 14:53:38,018 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-01 14:53:38,018 INFO [Listener at localhost/46667] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-01 14:53:38,018 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-01 14:53:38,018 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-01 14:53:38,172 INFO [Listener at localhost/46667] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-01 14:53:38,261 DEBUG [Listener at localhost/46667] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-01 14:53:38,360 INFO [Listener at localhost/46667] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35215 2023-04-01 14:53:38,371 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 14:53:38,373 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 14:53:38,395 INFO [Listener at localhost/46667] zookeeper.RecoverableZooKeeper(93): Process identifier=master:35215 connecting to ZooKeeper ensemble=127.0.0.1:60465 2023-04-01 14:53:38,436 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:352150x0, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-01 14:53:38,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:35215-0x101403728ea0000 connected 2023-04-01 14:53:38,465 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-01 14:53:38,466 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-01 14:53:38,469 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-01 14:53:38,478 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35215 2023-04-01 14:53:38,478 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35215 2023-04-01 14:53:38,483 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35215 2023-04-01 14:53:38,483 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35215 2023-04-01 14:53:38,484 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35215 2023-04-01 14:53:38,490 INFO [Listener at localhost/46667] master.HMaster(444): hbase.rootdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a, hbase.cluster.distributed=false 2023-04-01 14:53:38,568 INFO [Listener at localhost/46667] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-01 14:53:38,568 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-01 14:53:38,569 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-01 14:53:38,569 INFO [Listener at localhost/46667] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-01 14:53:38,569 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-01 14:53:38,569 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-01 14:53:38,574 INFO [Listener at localhost/46667] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-01 14:53:38,577 INFO [Listener at localhost/46667] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40863 2023-04-01 14:53:38,579 INFO [Listener at localhost/46667] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-01 14:53:38,587 DEBUG [Listener at localhost/46667] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-01 14:53:38,588 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 14:53:38,591 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 14:53:38,592 INFO [Listener at localhost/46667] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40863 connecting to ZooKeeper ensemble=127.0.0.1:60465 2023-04-01 14:53:38,596 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:408630x0, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-01 14:53:38,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40863-0x101403728ea0001 connected 2023-04-01 14:53:38,598 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-01 14:53:38,599 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-01 14:53:38,600 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-01 14:53:38,601 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40863 2023-04-01 14:53:38,601 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40863 2023-04-01 14:53:38,602 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40863 2023-04-01 14:53:38,602 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40863 2023-04-01 14:53:38,603 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40863 2023-04-01 14:53:38,606 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,35215,1680360817296 2023-04-01 14:53:38,618 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-01 14:53:38,620 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,35215,1680360817296 2023-04-01 14:53:38,639 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-01 14:53:38,639 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-01 14:53:38,639 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 14:53:38,640 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-01 14:53:38,641 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,35215,1680360817296 from backup master directory 2023-04-01 14:53:38,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-01 14:53:38,646 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,35215,1680360817296 2023-04-01 14:53:38,646 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-01 14:53:38,646 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-01 14:53:38,647 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,35215,1680360817296 2023-04-01 14:53:38,650 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-01 14:53:38,651 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-01 14:53:38,743 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/hbase.id with ID: 8e7addc7-e970-4936-8f35-13aaef06d6d3 2023-04-01 14:53:38,788 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-01 14:53:38,803 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 14:53:38,847 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x30d22915 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:38,882 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4606c96b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:38,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 14:53:38,909 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-01 14:53:38,927 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-01 14:53:38,927 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-01 14:53:38,929 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:53:38,933 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:53:38,934 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-01 14:53:38,968 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/data/master/store-tmp 2023-04-01 14:53:38,996 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:53:38,996 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-01 14:53:38,996 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 14:53:38,996 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 14:53:38,996 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-01 14:53:38,996 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 14:53:38,996 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 14:53:38,996 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-01 14:53:38,998 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/WALs/jenkins-hbase4.apache.org,35215,1680360817296 2023-04-01 14:53:39,022 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35215%2C1680360817296, suffix=, logDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/WALs/jenkins-hbase4.apache.org,35215,1680360817296, archiveDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/oldWALs, maxLogs=10 2023-04-01 14:53:39,075 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43281,DS-4d3588f6-7a66-4dcd-af22-6baef11c9ddb,DISK] 2023-04-01 14:53:39,083 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:53:39,142 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/WALs/jenkins-hbase4.apache.org,35215,1680360817296/jenkins-hbase4.apache.org%2C35215%2C1680360817296.1680360819032 2023-04-01 14:53:39,143 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43281,DS-4d3588f6-7a66-4dcd-af22-6baef11c9ddb,DISK]] 2023-04-01 14:53:39,144 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:53:39,144 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:53:39,148 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-01 14:53:39,149 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-01 14:53:39,205 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-01 14:53:39,213 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-01 14:53:39,236 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-01 14:53:39,248 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:53:39,254 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-01 14:53:39,256 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-01 14:53:39,271 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-01 14:53:39,275 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 14:53:39,277 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73431189, jitterRate=0.09420998394489288}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-01 14:53:39,277 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-01 14:53:39,278 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-01 14:53:39,297 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-01 14:53:39,298 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-01 14:53:39,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-01 14:53:39,302 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-01 14:53:39,334 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-04-01 14:53:39,335 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-01 14:53:39,360 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-01 14:53:39,365 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-01 14:53:39,389 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-01 14:53:39,392 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-01 14:53:39,394 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-01 14:53:39,399 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-01 14:53:39,403 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-01 14:53:39,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 14:53:39,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-01 14:53:39,408 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-01 14:53:39,421 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-01 14:53:39,427 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-01 14:53:39,427 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-01 14:53:39,427 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 14:53:39,428 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,35215,1680360817296, sessionid=0x101403728ea0000, setting cluster-up flag (Was=false) 2023-04-01 14:53:39,444 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 14:53:39,450 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-01 14:53:39,452 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35215,1680360817296 2023-04-01 14:53:39,461 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 14:53:39,468 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-01 14:53:39,469 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35215,1680360817296 2023-04-01 14:53:39,472 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.hbase-snapshot/.tmp 2023-04-01 14:53:39,509 INFO [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(952): ClusterId : 8e7addc7-e970-4936-8f35-13aaef06d6d3 2023-04-01 14:53:39,514 DEBUG [RS:0;jenkins-hbase4:40863] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-01 14:53:39,523 DEBUG [RS:0;jenkins-hbase4:40863] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-01 14:53:39,524 DEBUG [RS:0;jenkins-hbase4:40863] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-01 14:53:39,529 DEBUG [RS:0;jenkins-hbase4:40863] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-01 14:53:39,530 DEBUG [RS:0;jenkins-hbase4:40863] zookeeper.ReadOnlyZKClient(139): Connect 0x23e83673 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:39,536 DEBUG [RS:0;jenkins-hbase4:40863] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d2fdc7e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:39,537 DEBUG [RS:0;jenkins-hbase4:40863] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29105d03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-01 14:53:39,569 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:40863 2023-04-01 14:53:39,574 INFO [RS:0;jenkins-hbase4:40863] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-01 14:53:39,574 INFO [RS:0;jenkins-hbase4:40863] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-01 14:53:39,574 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1023): About to register with Master. 2023-04-01 14:53:39,578 INFO [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35215,1680360817296 with isa=jenkins-hbase4.apache.org/172.31.14.131:40863, startcode=1680360818567 2023-04-01 14:53:39,601 DEBUG [RS:0;jenkins-hbase4:40863] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-01 14:53:39,617 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-01 14:53:39,631 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-01 14:53:39,631 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-01 14:53:39,631 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-01 14:53:39,631 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-01 14:53:39,631 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-01 14:53:39,632 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-01 14:53:39,632 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-01 14:53:39,632 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-01 14:53:39,640 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680360849640 2023-04-01 14:53:39,643 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-01 14:53:39,645 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-01 14:53:39,646 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-01 14:53:39,653 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-01 14:53:39,656 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-01 14:53:39,664 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39115, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-01 14:53:39,668 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-01 14:53:39,669 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-01 14:53:39,669 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-01 14:53:39,669 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-01 14:53:39,670 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:39,673 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-01 14:53:39,676 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-01 14:53:39,677 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-01 14:53:39,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:39,680 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-01 14:53:39,681 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-01 14:53:39,688 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680360819683,5,FailOnTimeoutGroup] 2023-04-01 14:53:39,689 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680360819688,5,FailOnTimeoutGroup] 2023-04-01 14:53:39,689 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:39,689 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-01 14:53:39,691 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:39,693 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:39,713 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(2829): Master is not running yet 2023-04-01 14:53:39,714 WARN [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-04-01 14:53:39,815 INFO [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35215,1680360817296 with isa=jenkins-hbase4.apache.org/172.31.14.131:40863, startcode=1680360818567 2023-04-01 14:53:39,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:39,830 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a 2023-04-01 14:53:39,830 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39359 2023-04-01 14:53:39,830 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-01 14:53:39,835 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-01 14:53:39,837 DEBUG [RS:0;jenkins-hbase4:40863] zookeeper.ZKUtil(162): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:39,837 WARN [RS:0;jenkins-hbase4:40863] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-01 14:53:39,837 INFO [RS:0;jenkins-hbase4:40863] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-01 14:53:39,838 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/WALs/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:39,838 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40863,1680360818567] 2023-04-01 14:53:39,847 DEBUG [RS:0;jenkins-hbase4:40863] zookeeper.ZKUtil(162): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:39,858 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-01 14:53:39,866 INFO [RS:0;jenkins-hbase4:40863] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-01 14:53:39,885 INFO [RS:0;jenkins-hbase4:40863] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-01 14:53:39,890 INFO [RS:0;jenkins-hbase4:40863] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-01 14:53:39,891 INFO [RS:0;jenkins-hbase4:40863] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:39,892 INFO [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-01 14:53:39,901 INFO [RS:0;jenkins-hbase4:40863] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:39,901 DEBUG [RS:0;jenkins-hbase4:40863] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-01 14:53:39,902 DEBUG [RS:0;jenkins-hbase4:40863] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-01 14:53:39,902 DEBUG [RS:0;jenkins-hbase4:40863] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-01 14:53:39,902 DEBUG [RS:0;jenkins-hbase4:40863] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-01 14:53:39,902 DEBUG [RS:0;jenkins-hbase4:40863] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-01 14:53:39,902 DEBUG [RS:0;jenkins-hbase4:40863] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-01 14:53:39,902 DEBUG [RS:0;jenkins-hbase4:40863] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-01 14:53:39,903 DEBUG [RS:0;jenkins-hbase4:40863] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-01 14:53:39,903 DEBUG [RS:0;jenkins-hbase4:40863] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-01 14:53:39,903 DEBUG [RS:0;jenkins-hbase4:40863] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-01 14:53:39,905 INFO [RS:0;jenkins-hbase4:40863] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:39,905 INFO [RS:0;jenkins-hbase4:40863] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:39,905 INFO [RS:0;jenkins-hbase4:40863] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:39,921 INFO [RS:0;jenkins-hbase4:40863] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-01 14:53:39,924 INFO [RS:0;jenkins-hbase4:40863] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40863,1680360818567-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:39,941 INFO [RS:0;jenkins-hbase4:40863] regionserver.Replication(203): jenkins-hbase4.apache.org,40863,1680360818567 started 2023-04-01 14:53:39,941 INFO [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,40863,1680360818567, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40863, sessionid=0x101403728ea0001 2023-04-01 14:53:39,942 DEBUG [RS:0;jenkins-hbase4:40863] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-01 14:53:39,942 DEBUG [RS:0;jenkins-hbase4:40863] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:39,942 DEBUG [RS:0;jenkins-hbase4:40863] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40863,1680360818567' 2023-04-01 14:53:39,942 DEBUG [RS:0;jenkins-hbase4:40863] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:39,943 DEBUG [RS:0;jenkins-hbase4:40863] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:39,944 DEBUG [RS:0;jenkins-hbase4:40863] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-01 14:53:39,944 DEBUG [RS:0;jenkins-hbase4:40863] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-01 14:53:39,944 DEBUG [RS:0;jenkins-hbase4:40863] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:39,944 DEBUG [RS:0;jenkins-hbase4:40863] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40863,1680360818567' 2023-04-01 14:53:39,944 DEBUG [RS:0;jenkins-hbase4:40863] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-01 14:53:39,944 DEBUG [RS:0;jenkins-hbase4:40863] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-01 14:53:39,945 DEBUG [RS:0;jenkins-hbase4:40863] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-01 14:53:39,945 INFO [RS:0;jenkins-hbase4:40863] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-01 14:53:39,945 INFO [RS:0;jenkins-hbase4:40863] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-01 14:53:40,058 INFO [RS:0;jenkins-hbase4:40863] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40863%2C1680360818567, suffix=, logDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/WALs/jenkins-hbase4.apache.org,40863,1680360818567, archiveDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/oldWALs, maxLogs=32 2023-04-01 14:53:40,078 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43281,DS-4d3588f6-7a66-4dcd-af22-6baef11c9ddb,DISK] 2023-04-01 14:53:40,084 INFO [RS:0;jenkins-hbase4:40863] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/WALs/jenkins-hbase4.apache.org,40863,1680360818567/jenkins-hbase4.apache.org%2C40863%2C1680360818567.1680360820060 2023-04-01 14:53:40,085 DEBUG [RS:0;jenkins-hbase4:40863] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43281,DS-4d3588f6-7a66-4dcd-af22-6baef11c9ddb,DISK]] 2023-04-01 14:53:40,102 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-01 14:53:40,103 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-01 14:53:40,103 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a 2023-04-01 14:53:40,524 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:53:40,527 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-01 14:53:40,530 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/info 2023-04-01 14:53:40,531 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-01 14:53:40,532 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:53:40,532 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-01 14:53:40,534 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/rep_barrier 2023-04-01 14:53:40,535 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-01 14:53:40,536 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:53:40,536 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-01 14:53:40,538 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/table 2023-04-01 14:53:40,539 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-01 14:53:40,539 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:53:40,541 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740 2023-04-01 14:53:40,542 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740 2023-04-01 14:53:40,546 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 14:53:40,547 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-01 14:53:40,551 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 14:53:40,552 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59871031, jitterRate=-0.10785211622714996}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 14:53:40,552 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-01 14:53:40,552 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-01 14:53:40,552 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-01 14:53:40,552 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-01 14:53:40,552 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-01 14:53:40,552 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-01 14:53:40,553 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-01 14:53:40,553 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-01 14:53:40,559 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-01 14:53:40,559 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-01 14:53:40,567 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-01 14:53:40,581 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-01 14:53:40,584 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-01 14:53:40,736 DEBUG [jenkins-hbase4:35215] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-01 14:53:40,739 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40863,1680360818567, state=OPENING 2023-04-01 14:53:40,746 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-01 14:53:40,748 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 14:53:40,748 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-01 14:53:40,751 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:53:40,933 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:40,936 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-01 14:53:40,939 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40518, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-01 14:53:40,949 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-01 14:53:40,950 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-01 14:53:40,953 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40863%2C1680360818567.meta, suffix=.meta, logDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/WALs/jenkins-hbase4.apache.org,40863,1680360818567, archiveDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/oldWALs, maxLogs=32 2023-04-01 14:53:40,968 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43281,DS-4d3588f6-7a66-4dcd-af22-6baef11c9ddb,DISK] 2023-04-01 14:53:40,972 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/WALs/jenkins-hbase4.apache.org,40863,1680360818567/jenkins-hbase4.apache.org%2C40863%2C1680360818567.meta.1680360820954.meta 2023-04-01 14:53:40,973 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43281,DS-4d3588f6-7a66-4dcd-af22-6baef11c9ddb,DISK]] 2023-04-01 14:53:40,973 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:53:40,975 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-01 14:53:40,997 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-01 14:53:41,003 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-01 14:53:41,009 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-01 14:53:41,009 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:53:41,009 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-01 14:53:41,010 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-01 14:53:41,013 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-01 14:53:41,016 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/info 2023-04-01 14:53:41,016 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/info 2023-04-01 14:53:41,017 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-01 14:53:41,018 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:53:41,018 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-01 14:53:41,020 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/rep_barrier 2023-04-01 14:53:41,020 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/rep_barrier 2023-04-01 14:53:41,021 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-01 14:53:41,022 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:53:41,022 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-01 14:53:41,023 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/table 2023-04-01 14:53:41,024 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/table 2023-04-01 14:53:41,024 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-01 14:53:41,025 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:53:41,027 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740 2023-04-01 14:53:41,030 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740 2023-04-01 14:53:41,034 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 14:53:41,037 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-01 14:53:41,038 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66656122, jitterRate=-0.006746381521224976}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 14:53:41,038 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-01 14:53:41,049 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680360820925 2023-04-01 14:53:41,066 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-01 14:53:41,066 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-01 14:53:41,067 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40863,1680360818567, state=OPEN 2023-04-01 14:53:41,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-01 14:53:41,070 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-01 14:53:41,073 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-01 14:53:41,073 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40863,1680360818567 in 320 msec 2023-04-01 14:53:41,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-01 14:53:41,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 507 msec 2023-04-01 14:53:41,083 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5620 sec 2023-04-01 14:53:41,083 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680360821083, completionTime=-1 2023-04-01 14:53:41,083 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-01 14:53:41,083 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-01 14:53:41,144 DEBUG [hconnection-0x4f5d57d6-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:41,148 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40534, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:41,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-01 14:53:41,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680360881168 2023-04-01 14:53:41,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680360941169 2023-04-01 14:53:41,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 85 msec 2023-04-01 14:53:41,189 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35215,1680360817296-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:41,190 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35215,1680360817296-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:41,190 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35215,1680360817296-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:41,191 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:35215, period=300000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:41,192 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-01 14:53:41,197 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-01 14:53:41,206 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-01 14:53:41,207 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-01 14:53:41,217 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-01 14:53:41,220 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 14:53:41,222 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 14:53:41,243 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1 2023-04-01 14:53:41,246 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1 empty. 2023-04-01 14:53:41,246 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1 2023-04-01 14:53:41,247 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-01 14:53:41,672 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-01 14:53:41,675 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 258dfbc10657b5aa46b7285094b199c1, NAME => 'hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp 2023-04-01 14:53:41,690 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:53:41,690 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 258dfbc10657b5aa46b7285094b199c1, disabling compactions & flushes 2023-04-01 14:53:41,691 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:53:41,691 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:53:41,691 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. after waiting 0 ms 2023-04-01 14:53:41,691 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:53:41,691 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:53:41,691 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 258dfbc10657b5aa46b7285094b199c1: 2023-04-01 14:53:41,696 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 14:53:41,714 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680360821699"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360821699"}]},"ts":"1680360821699"} 2023-04-01 14:53:41,745 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 14:53:41,747 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 14:53:41,752 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360821747"}]},"ts":"1680360821747"} 2023-04-01 14:53:41,756 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-01 14:53:41,766 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=258dfbc10657b5aa46b7285094b199c1, ASSIGN}] 2023-04-01 14:53:41,768 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=258dfbc10657b5aa46b7285094b199c1, ASSIGN 2023-04-01 14:53:41,769 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=258dfbc10657b5aa46b7285094b199c1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40863,1680360818567; forceNewPlan=false, retain=false 2023-04-01 14:53:41,920 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=258dfbc10657b5aa46b7285094b199c1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:41,921 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680360821920"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360821920"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360821920"}]},"ts":"1680360821920"} 2023-04-01 14:53:41,925 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 258dfbc10657b5aa46b7285094b199c1, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:53:42,085 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:53:42,086 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 258dfbc10657b5aa46b7285094b199c1, NAME => 'hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1.', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:53:42,087 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 258dfbc10657b5aa46b7285094b199c1 2023-04-01 14:53:42,088 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:53:42,088 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 258dfbc10657b5aa46b7285094b199c1 2023-04-01 14:53:42,088 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 258dfbc10657b5aa46b7285094b199c1 2023-04-01 14:53:42,090 INFO [StoreOpener-258dfbc10657b5aa46b7285094b199c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 258dfbc10657b5aa46b7285094b199c1 2023-04-01 14:53:42,093 DEBUG [StoreOpener-258dfbc10657b5aa46b7285094b199c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1/info 2023-04-01 14:53:42,093 DEBUG [StoreOpener-258dfbc10657b5aa46b7285094b199c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1/info 2023-04-01 14:53:42,093 INFO [StoreOpener-258dfbc10657b5aa46b7285094b199c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 258dfbc10657b5aa46b7285094b199c1 columnFamilyName info 2023-04-01 14:53:42,094 INFO [StoreOpener-258dfbc10657b5aa46b7285094b199c1-1] regionserver.HStore(310): Store=258dfbc10657b5aa46b7285094b199c1/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:53:42,096 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1 2023-04-01 14:53:42,096 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1 2023-04-01 14:53:42,101 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 258dfbc10657b5aa46b7285094b199c1 2023-04-01 14:53:42,105 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 14:53:42,106 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 258dfbc10657b5aa46b7285094b199c1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65996366, jitterRate=-0.01657751202583313}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-01 14:53:42,106 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 258dfbc10657b5aa46b7285094b199c1: 2023-04-01 14:53:42,108 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1., pid=6, masterSystemTime=1680360822079 2023-04-01 14:53:42,112 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:53:42,112 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:53:42,113 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=258dfbc10657b5aa46b7285094b199c1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:42,114 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680360822112"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680360822112"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680360822112"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680360822112"}]},"ts":"1680360822112"} 2023-04-01 14:53:42,120 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-01 14:53:42,120 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 258dfbc10657b5aa46b7285094b199c1, server=jenkins-hbase4.apache.org,40863,1680360818567 in 192 msec 2023-04-01 14:53:42,123 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-01 14:53:42,124 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=258dfbc10657b5aa46b7285094b199c1, ASSIGN in 354 msec 2023-04-01 14:53:42,125 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 14:53:42,126 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360822126"}]},"ts":"1680360822126"} 2023-04-01 14:53:42,129 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-01 14:53:42,133 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 14:53:42,136 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 925 msec 2023-04-01 14:53:42,220 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-01 14:53:42,223 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-01 14:53:42,223 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 14:53:42,264 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-01 14:53:42,282 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-01 14:53:42,288 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 35 msec 2023-04-01 14:53:42,298 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-01 14:53:42,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-01 14:53:42,314 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-04-01 14:53:42,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-01 14:53:42,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-01 14:53:42,329 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.682sec 2023-04-01 14:53:42,330 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-01 14:53:42,332 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-01 14:53:42,332 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-01 14:53:42,333 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35215,1680360817296-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-01 14:53:42,334 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35215,1680360817296-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-01 14:53:42,344 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-01 14:53:42,414 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x3ddd91d3 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:42,418 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51f71277, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:42,431 DEBUG [hconnection-0x286751c0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:42,443 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40548, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:42,453 INFO [Listener at localhost/46667] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,35215,1680360817296 2023-04-01 14:53:42,472 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=261, ProcessCount=170, AvailableMemoryMB=4150 2023-04-01 14:53:42,489 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 14:53:42,492 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49138, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 14:53:42,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 14:53:42,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 14:53:42,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 14:53:42,509 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 14:53:42,511 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 14:53:42,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-01 14:53:42,514 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:42,516 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3 empty. 2023-04-01 14:53:42,517 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:42,518 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:53:42,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-01 14:53:42,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-01 14:53:42,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-01 14:53:42,936 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 14:53:42,938 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8f567672fffd4bc77debf33ecd862db3, NAME => 'TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp 2023-04-01 14:53:42,951 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:53:42,951 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8f567672fffd4bc77debf33ecd862db3, disabling compactions & flushes 2023-04-01 14:53:42,951 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:42,951 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:42,951 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. after waiting 0 ms 2023-04-01 14:53:42,951 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:42,951 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:42,951 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:42,956 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 14:53:42,957 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360822957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360822957"}]},"ts":"1680360822957"} 2023-04-01 14:53:42,960 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 14:53:42,962 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 14:53:42,963 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360822962"}]},"ts":"1680360822962"} 2023-04-01 14:53:42,965 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 14:53:42,970 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8f567672fffd4bc77debf33ecd862db3, ASSIGN}] 2023-04-01 14:53:42,972 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8f567672fffd4bc77debf33ecd862db3, ASSIGN 2023-04-01 14:53:42,974 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8f567672fffd4bc77debf33ecd862db3, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40863,1680360818567; forceNewPlan=false, retain=false 2023-04-01 14:53:43,126 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=8f567672fffd4bc77debf33ecd862db3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:43,126 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360823125"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360823125"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360823125"}]},"ts":"1680360823125"} 2023-04-01 14:53:43,129 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:53:43,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-01 14:53:43,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:43,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8f567672fffd4bc77debf33ecd862db3, NAME => 'TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:53:43,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:43,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:53:43,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:43,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:43,291 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:43,294 DEBUG [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A 2023-04-01 14:53:43,294 DEBUG [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A 2023-04-01 14:53:43,295 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:53:43,295 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8f567672fffd4bc77debf33ecd862db3 columnFamilyName A 2023-04-01 14:53:43,296 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] regionserver.HStore(310): Store=8f567672fffd4bc77debf33ecd862db3/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:53:43,296 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:43,298 DEBUG [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B 2023-04-01 14:53:43,298 DEBUG [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B 2023-04-01 14:53:43,299 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:53:43,299 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8f567672fffd4bc77debf33ecd862db3 columnFamilyName B 2023-04-01 14:53:43,300 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] regionserver.HStore(310): Store=8f567672fffd4bc77debf33ecd862db3/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:53:43,300 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:43,302 DEBUG [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C 2023-04-01 14:53:43,302 DEBUG [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C 2023-04-01 14:53:43,302 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:53:43,302 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8f567672fffd4bc77debf33ecd862db3 columnFamilyName C 2023-04-01 14:53:43,303 INFO [StoreOpener-8f567672fffd4bc77debf33ecd862db3-1] regionserver.HStore(310): Store=8f567672fffd4bc77debf33ecd862db3/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:53:43,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:43,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:43,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:43,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 14:53:43,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:43,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 14:53:43,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8f567672fffd4bc77debf33ecd862db3; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74977006, jitterRate=0.11724445223808289}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 14:53:43,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:43,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., pid=11, masterSystemTime=1680360823282 2023-04-01 14:53:43,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:43,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:43,321 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=8f567672fffd4bc77debf33ecd862db3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:43,322 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360823321"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680360823321"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680360823321"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680360823321"}]},"ts":"1680360823321"} 2023-04-01 14:53:43,327 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-01 14:53:43,327 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 in 195 msec 2023-04-01 14:53:43,330 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-01 14:53:43,331 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8f567672fffd4bc77debf33ecd862db3, ASSIGN in 357 msec 2023-04-01 14:53:43,332 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 14:53:43,332 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360823332"}]},"ts":"1680360823332"} 2023-04-01 14:53:43,334 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 14:53:43,340 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 14:53:43,342 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 835 msec 2023-04-01 14:53:43,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-01 14:53:43,635 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-01 14:53:43,640 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x44ef43e3 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:43,645 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2678d495, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:43,648 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:43,654 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:43,657 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 14:53:43,659 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57746, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 14:53:43,667 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x61e72c30 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:43,672 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b886c8f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:43,673 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x56c8e45e to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:43,677 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56223d42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:43,677 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x4fea27db to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:43,681 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77648431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:43,681 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x6038a026 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:43,686 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de4d1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:43,687 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x63859c62 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:43,692 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9a575b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:43,693 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x49a45b6d to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:43,696 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1818ad0a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:43,697 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x6fef7a21 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:43,715 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fb2b82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:43,716 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd3abbd to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:43,719 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6129209e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:43,720 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x6aad2e18 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:53:43,761 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b02eeb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:53:43,788 DEBUG [hconnection-0x2b02ff72-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:43,793 DEBUG [hconnection-0x11a56528-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:43,794 DEBUG [hconnection-0x29bd923d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:43,795 DEBUG [hconnection-0x2035e7e9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:43,801 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:43,795 DEBUG [hconnection-0x39df0911-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:43,801 DEBUG [hconnection-0x30cf85db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:43,806 DEBUG [hconnection-0x2e0f9076-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:43,806 DEBUG [hconnection-0x2faf6a30-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:43,807 DEBUG [hconnection-0xd322817-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:53:43,807 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:43,808 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51660, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:43,816 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51672, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:43,816 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:43,824 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51686, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:43,826 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51702, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:43,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:43,835 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51712, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:43,836 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51728, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:53:43,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:43,857 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:43,857 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:43,858 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:43,858 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:43,858 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:43,858 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:43,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:43,860 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:43,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:43,860 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:43,860 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:43,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:43,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:43,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:43,867 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:43,868 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:43,868 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:43,869 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:43,869 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:43,870 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-01 14:53:43,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:43,878 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:43,896 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:43,897 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:43,897 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:43,897 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:43,897 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:43,898 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:44,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680360823974, max timestamp=1680360823994Num uniques 859; 2023-04-01 14:53:44,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680360823974, max timestamp=1680360823994Num uniques 859; 2023-04-01 14:53:44,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 14:53:44,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 14:53:44,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680360823974, max timestamp=1680360823994Num uniques 859; 2023-04-01 14:53:44,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 14:53:44,035 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/13ac35b3a6b840688c4e8d8353c3ec67 2023-04-01 14:53:44,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360884058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360884062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680360824043, max timestamp=1680360824068Num uniques 1009; 2023-04-01 14:53:44,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-04-01 14:53:44,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360884089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360884097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360884100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680360824043, max timestamp=1680360824068Num uniques 1009; 2023-04-01 14:53:44,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-01 14:53:44,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680360824043, max timestamp=1680360824070Num uniques 1009; 2023-04-01 14:53:44,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-01 14:53:44,164 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/d75019da49664d0b9f8ef7d7cd9bad92 2023-04-01 14:53:44,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360884226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360884226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360884227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360884243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360884248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,290 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/b9aed4cc7bd2485c9bfc119415ae933c 2023-04-01 14:53:44,303 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/13ac35b3a6b840688c4e8d8353c3ec67 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/13ac35b3a6b840688c4e8d8353c3ec67 2023-04-01 14:53:44,316 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/13ac35b3a6b840688c4e8d8353c3ec67, entries=150, sequenceid=11, filesize=11.6 K 2023-04-01 14:53:44,327 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/d75019da49664d0b9f8ef7d7cd9bad92 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/d75019da49664d0b9f8ef7d7cd9bad92 2023-04-01 14:53:44,367 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/d75019da49664d0b9f8ef7d7cd9bad92, entries=150, sequenceid=11, filesize=11.6 K 2023-04-01 14:53:44,374 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/b9aed4cc7bd2485c9bfc119415ae933c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b9aed4cc7bd2485c9bfc119415ae933c 2023-04-01 14:53:44,423 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b9aed4cc7bd2485c9bfc119415ae933c, entries=150, sequenceid=11, filesize=11.6 K 2023-04-01 14:53:44,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,429 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.82 KB/138054 for 8f567672fffd4bc77debf33ecd862db3 in 559ms, sequenceid=11, compaction requested=false 2023-04-01 14:53:44,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360884353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,436 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-01 14:53:44,439 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:44,439 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,449 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:44,449 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:44,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:44,459 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,459 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,459 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:44,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680360824212, max timestamp=1680360824436Num uniques 1077; 2023-04-01 14:53:44,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-01 14:53:44,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:44,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=114.69 KB heapSize=327.55 KB 2023-04-01 14:53:44,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680360824212, max timestamp=1680360824436Num uniques 1009; 2023-04-01 14:53:44,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-04-01 14:53:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:44,469 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:44,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:44,473 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:44,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18ed2eaf[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:44,473 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:44,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:44,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:44,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:44,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 14:53:44,478 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,478 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:44,478 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:44,478 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:44,483 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:44,484 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:44,484 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:44,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,504 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:44,505 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:44,505 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:44,505 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:44,505 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:44,505 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:44,505 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:44,513 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,513 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,513 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:44,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:44,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:44,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:44,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:44,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,516 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:44,523 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:44,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:44,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:44,588 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:44,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:44,588 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:44,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:44,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:44,588 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:44,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:44,592 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,593 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,593 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:44,594 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:44,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/61edd2acaed64d9bb367e6f6edecb1e5 2023-04-01 14:53:44,659 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/35a6fd30b8dd4a4581d7a9b3f94b2424 2023-04-01 14:53:44,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360884665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360884662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360884668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,694 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:44,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:44,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360884687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360884692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,716 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=50.64 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/9482ec81074c45bea68019193d5c6994 2023-04-01 14:53:44,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:44,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/61edd2acaed64d9bb367e6f6edecb1e5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61edd2acaed64d9bb367e6f6edecb1e5 2023-04-01 14:53:44,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:44,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:44,735 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:44,736 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:44,745 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:44,745 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:44,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:44,748 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:44,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:44,748 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:44,749 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:44,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:44,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:44,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61edd2acaed64d9bb367e6f6edecb1e5, entries=300, sequenceid=68, filesize=18.5 K 2023-04-01 14:53:44,762 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/35a6fd30b8dd4a4581d7a9b3f94b2424 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/35a6fd30b8dd4a4581d7a9b3f94b2424 2023-04-01 14:53:44,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:44,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:44,764 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:44,764 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:44,766 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:44,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:44,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:44,771 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:44,771 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:44,772 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:44,772 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:44,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:44,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:44,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:44,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7faa551b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:44,786 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:44,786 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,789 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:44,790 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,790 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:44,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/35a6fd30b8dd4a4581d7a9b3f94b2424, entries=150, sequenceid=68, filesize=11.6 K 2023-04-01 14:53:44,797 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:44,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,797 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:44,798 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:44,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/9482ec81074c45bea68019193d5c6994 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9482ec81074c45bea68019193d5c6994 2023-04-01 14:53:44,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:44,806 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:44,806 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:44,806 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:44,806 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:44,806 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:44,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360884790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360884791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360884800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:44,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:44,814 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9482ec81074c45bea68019193d5c6994, entries=150, sequenceid=68, filesize=11.6 K 2023-04-01 14:53:44,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,815 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~77.47 KB/79332, heapSize ~215.92 KB/221104, currentSize=107.34 KB/109920 for 8f567672fffd4bc77debf33ecd862db3 in 347ms, sequenceid=68, compaction requested=false 2023-04-01 14:53:44,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:44,823 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:44,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:44,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:44,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:44,831 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:44,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:44,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:44,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:44,836 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,836 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,836 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:44,837 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:44,837 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:44,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:44,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:44,838 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=114.05 KB heapSize=339.66 KB 2023-04-01 14:53:44,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:44,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:44,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:44,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:44,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:44,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:44,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:44,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/bad19c79e21d4ce28b5926803ef14b13 2023-04-01 14:53:44,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360884897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:44,937 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:44,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360884915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:44,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:44,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:44,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:44,950 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:44,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:44,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:44,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:44,952 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,952 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:44,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:44,953 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:44,955 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,955 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:44,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:44,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:44,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:44,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:44,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:44,956 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:44,956 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:44,958 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:44,958 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:44,958 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:44,960 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:44,960 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:44,960 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:44,960 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:44,963 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:44,963 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:44,963 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:44,971 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:44,972 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:44,973 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,973 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:44,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:44,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57b408a7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:44,975 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:44,975 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,976 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:44,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:44,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:44,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:44,977 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,977 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:44,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:44,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:44,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:44,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:44,990 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:44,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:44,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:44,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:44,991 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:44,993 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,993 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:44,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,993 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:44,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:44,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:44,997 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,006 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:45,006 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,006 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:45,006 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:45,006 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:45,008 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:45,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:45,008 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:45,009 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:45,010 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:45,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:45,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360885013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360885013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360885014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360885033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360885040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,110 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:45,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:45,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:45,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:45,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:45,128 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:45,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:45,129 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:45,129 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:45,130 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,130 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,131 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:45,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:45,133 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:45,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:45,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:45,139 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:45,140 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:45,148 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:45,148 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:45,148 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:45,148 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:45,150 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:45,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,162 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:45,163 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,163 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@407a4f18[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:45,166 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:45,166 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,168 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:45,168 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:45,169 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,169 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:45,171 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:45,171 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,171 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:45,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:45,177 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:45,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:45,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:45,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:45,177 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:45,180 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,180 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:45,180 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:45,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:45,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,184 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,187 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:45,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:45,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:45,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:45,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:45,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:45,194 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:45,194 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:45,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,194 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:45,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:45,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:45,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:45,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360885244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360885252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,295 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:45,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:45,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:45,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:45,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:45,312 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:45,312 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:45,313 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:45,313 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:45,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:45,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:45,316 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,316 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,317 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:45,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360885319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:45,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360885319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:45,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:45,321 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:45,322 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:45,322 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:45,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360885321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,324 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:45,324 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:45,326 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:45,326 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:45,326 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:45,326 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:45,331 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:45,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:45,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:45,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:45,335 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,335 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:45,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:45,335 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,336 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:45,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:45,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311389ee[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:45,339 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:45,339 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,340 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:45,341 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,341 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:45,343 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:45,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:45,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:45,345 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/48ff1771ac384a14bf543a872b3fbf1b 2023-04-01 14:53:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:45,351 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:45,351 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:45,351 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:45,351 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:45,351 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:45,356 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,356 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:45,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,356 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:45,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:45,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:45,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,359 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,362 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,369 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,369 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,370 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:45,370 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:45,370 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:45,370 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,370 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:45,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:45,370 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:45,370 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:45,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:45,370 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,371 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:45,371 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:45,405 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/cbca22a408f74e998217a59eb9ee28f6 2023-04-01 14:53:45,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/bad19c79e21d4ce28b5926803ef14b13 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bad19c79e21d4ce28b5926803ef14b13 2023-04-01 14:53:45,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bad19c79e21d4ce28b5926803ef14b13, entries=250, sequenceid=89, filesize=16.2 K 2023-04-01 14:53:45,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/48ff1771ac384a14bf543a872b3fbf1b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/48ff1771ac384a14bf543a872b3fbf1b 2023-04-01 14:53:45,444 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/48ff1771ac384a14bf543a872b3fbf1b, entries=150, sequenceid=89, filesize=11.6 K 2023-04-01 14:53:45,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/cbca22a408f74e998217a59eb9ee28f6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cbca22a408f74e998217a59eb9ee28f6 2023-04-01 14:53:45,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cbca22a408f74e998217a59eb9ee28f6, entries=150, sequenceid=89, filesize=11.6 K 2023-04-01 14:53:45,468 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 8f567672fffd4bc77debf33ecd862db3 in 630ms, sequenceid=89, compaction requested=true 2023-04-01 14:53:45,469 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:45,472 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:45,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:45,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:45,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:45,476 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:53:45,477 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:53:45,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:45,481 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:53:45,484 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/B is initiating minor compaction (all files) 2023-04-01 14:53:45,485 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/B in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:45,485 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/d75019da49664d0b9f8ef7d7cd9bad92, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/35a6fd30b8dd4a4581d7a9b3f94b2424, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/48ff1771ac384a14bf543a872b3fbf1b] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=34.8 K 2023-04-01 14:53:45,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:45,489 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting d75019da49664d0b9f8ef7d7cd9bad92, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680360823844 2023-04-01 14:53:45,493 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 47358 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:53:45,493 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/A is initiating minor compaction (all files) 2023-04-01 14:53:45,493 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/A in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:45,493 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/13ac35b3a6b840688c4e8d8353c3ec67, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61edd2acaed64d9bb367e6f6edecb1e5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bad19c79e21d4ce28b5926803ef14b13] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=46.2 K 2023-04-01 14:53:45,494 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 35a6fd30b8dd4a4581d7a9b3f94b2424, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680360824436 2023-04-01 14:53:45,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:45,495 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 13ac35b3a6b840688c4e8d8353c3ec67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680360823844 2023-04-01 14:53:45,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:45,496 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:45,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:45,496 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 48ff1771ac384a14bf543a872b3fbf1b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680360824682 2023-04-01 14:53:45,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:45,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:45,498 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,498 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,498 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 61edd2acaed64d9bb367e6f6edecb1e5, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680360824212 2023-04-01 14:53:45,500 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:45,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:45,500 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting bad19c79e21d4ce28b5926803ef14b13, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680360824653 2023-04-01 14:53:45,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:45,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:45,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:45,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:45,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:45,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:45,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:45,507 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:45,507 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:45,507 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:45,507 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-01 14:53:45,508 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:45,508 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:45,508 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:45,508 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:45,508 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:45,508 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:45,548 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#B#compaction#10 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:45,557 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#A#compaction#11 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:45,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:45,563 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:45,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360825641, max timestamp=1680360825660Num uniques 859; 2023-04-01 14:53:45,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:53:45,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360885678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360885680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360885786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360885787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360885823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360885827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360885830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,862 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-01 14:53:45,865 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-01 14:53:45,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360885989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:45,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:45,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360885991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,051 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/4cb570775fb04655b4b3d25cb16e2117 2023-04-01 14:53:46,089 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/2ab6282b23f34da488dba7de696a6003 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/2ab6282b23f34da488dba7de696a6003 2023-04-01 14:53:46,111 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/34211a328a124e9d99ba7fa32af71aba as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/34211a328a124e9d99ba7fa32af71aba 2023-04-01 14:53:46,138 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/B of 8f567672fffd4bc77debf33ecd862db3 into 34211a328a124e9d99ba7fa32af71aba(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:46,138 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:46,138 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/B, priority=13, startTime=1680360825476; duration=0sec 2023-04-01 14:53:46,139 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/A of 8f567672fffd4bc77debf33ecd862db3 into 2ab6282b23f34da488dba7de696a6003(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:46,140 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:46,140 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/A, priority=13, startTime=1680360825471; duration=0sec 2023-04-01 14:53:46,141 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:46,141 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:46,141 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:53:46,146 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:53:46,146 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/C is initiating minor compaction (all files) 2023-04-01 14:53:46,147 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/C in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:46,147 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b9aed4cc7bd2485c9bfc119415ae933c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9482ec81074c45bea68019193d5c6994, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cbca22a408f74e998217a59eb9ee28f6] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=34.8 K 2023-04-01 14:53:46,149 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b9aed4cc7bd2485c9bfc119415ae933c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680360823844 2023-04-01 14:53:46,149 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/342e06fa898140018c1f1eac36448297 2023-04-01 14:53:46,150 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9482ec81074c45bea68019193d5c6994, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680360824436 2023-04-01 14:53:46,151 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cbca22a408f74e998217a59eb9ee28f6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680360824682 2023-04-01 14:53:46,181 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#C#compaction#14 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:46,229 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/e467c815dace4c0197416ee3458efbed 2023-04-01 14:53:46,249 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/4cb570775fb04655b4b3d25cb16e2117 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/4cb570775fb04655b4b3d25cb16e2117 2023-04-01 14:53:46,257 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/707ee18b5131436e880a0f0a2428f9f5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/707ee18b5131436e880a0f0a2428f9f5 2023-04-01 14:53:46,267 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/4cb570775fb04655b4b3d25cb16e2117, entries=150, sequenceid=100, filesize=11.6 K 2023-04-01 14:53:46,274 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/342e06fa898140018c1f1eac36448297 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/342e06fa898140018c1f1eac36448297 2023-04-01 14:53:46,287 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/C of 8f567672fffd4bc77debf33ecd862db3 into 707ee18b5131436e880a0f0a2428f9f5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:46,288 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:46,288 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/C, priority=13, startTime=1680360825476; duration=0sec 2023-04-01 14:53:46,288 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:46,290 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/342e06fa898140018c1f1eac36448297, entries=150, sequenceid=100, filesize=11.6 K 2023-04-01 14:53:46,297 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/e467c815dace4c0197416ee3458efbed as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/e467c815dace4c0197416ee3458efbed 2023-04-01 14:53:46,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:46,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360886298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:46,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360886308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,321 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/e467c815dace4c0197416ee3458efbed, entries=150, sequenceid=100, filesize=11.6 K 2023-04-01 14:53:46,323 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=123.00 KB/125950 for 8f567672fffd4bc77debf33ecd862db3 in 816ms, sequenceid=100, compaction requested=false 2023-04-01 14:53:46,323 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:46,323 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:46,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:46,324 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:46,324 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,324 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:46,324 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:46,326 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:46,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:46,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,327 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:46,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:46,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:46,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cbc105a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:46,330 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:46,330 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,338 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,338 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:46,338 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:46,338 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:46,339 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,339 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:46,340 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:46,340 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:46,340 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:46,341 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:46,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:46,347 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:46,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:46,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:46,348 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:46,348 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:46,353 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,353 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:46,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,353 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:46,354 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:46,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:46,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:46,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,357 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:46,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:46,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:46,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:46,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:46,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:46,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:46,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:46,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:46,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:46,365 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:46,365 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:46,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:46,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:46,465 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:46,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:46,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:46,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:46,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:46,477 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:46,477 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:46,478 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:46,478 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:46,479 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,479 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:46,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:46,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:46,482 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:46,483 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,483 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:46,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:46,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:46,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:46,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:46,497 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:46,497 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:46,497 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:46,497 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:46,498 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:46,498 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=123.00 KB heapSize=361.72 KB 2023-04-01 14:53:46,498 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:46,498 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:46,499 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:46,499 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:46,499 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:46,499 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:46,576 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/9641c33393e8466094455e59b19d0434 2023-04-01 14:53:46,578 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 14:53:46,643 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/6913081a186443f68cc3649a50f4c3b3 2023-04-01 14:53:46,699 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/33854bc7845b466b914d098f9b697875 2023-04-01 14:53:46,711 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/9641c33393e8466094455e59b19d0434 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9641c33393e8466094455e59b19d0434 2023-04-01 14:53:46,721 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9641c33393e8466094455e59b19d0434, entries=150, sequenceid=129, filesize=11.6 K 2023-04-01 14:53:46,727 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/6913081a186443f68cc3649a50f4c3b3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6913081a186443f68cc3649a50f4c3b3 2023-04-01 14:53:46,735 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6913081a186443f68cc3649a50f4c3b3, entries=150, sequenceid=129, filesize=11.6 K 2023-04-01 14:53:46,745 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/33854bc7845b466b914d098f9b697875 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/33854bc7845b466b914d098f9b697875 2023-04-01 14:53:46,759 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/33854bc7845b466b914d098f9b697875, entries=150, sequenceid=129, filesize=11.6 K 2023-04-01 14:53:46,763 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~361.67 KB/370352, currentSize=0 B/0 for 8f567672fffd4bc77debf33ecd862db3 in 265ms, sequenceid=129, compaction requested=true 2023-04-01 14:53:46,763 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:46,763 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:46,764 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:46,764 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,764 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:46,764 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:46,768 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:46,769 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:46,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:46,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bbad25a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:46,772 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:46,772 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,774 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,774 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:46,775 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:46,775 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:46,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:46,778 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:46,779 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:46,779 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:46,779 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:46,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:46,785 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:46,785 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:46,785 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:46,785 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:46,785 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:46,787 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:46,788 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,788 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:46,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:46,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:46,789 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,793 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:46,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:46,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:46,797 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:46,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:46,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:46,806 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:46,806 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:46,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,806 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:46,806 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:46,807 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,807 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,807 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,807 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:46,848 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:53:46,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:46,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:46,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:46,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:46,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:46,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:46,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:46,906 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:46,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:46,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/158a1470a5f943f5b6f934cca449ec3d 2023-04-01 14:53:46,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:46,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:46,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360886925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:46,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360826881, max timestamp=1680360826936Num uniques 859; 2023-04-01 14:53:46,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:46,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:53:46,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:46,949 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:46,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:46,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360826881, max timestamp=1680360826936Num uniques 859; 2023-04-01 14:53:46,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:53:46,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360886933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360826881, max timestamp=1680360826936Num uniques 859; 2023-04-01 14:53:46,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:46,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:53:46,952 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:46,953 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,953 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:46,956 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,956 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:46,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:46,956 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,956 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:46,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:46,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:46,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:46,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:46,965 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:46,965 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:46,969 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:46,970 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:46,970 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:46,970 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:46,970 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:46,970 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:46,970 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:46,976 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:46,977 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,977 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:46,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:46,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@792a42ba[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:46,981 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:46,981 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/5711a6e25e664fb3a5867f3e3faff1a8 2023-04-01 14:53:46,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:46,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:46,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:46,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:46,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:46,985 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:46,986 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:46,986 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:46,986 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:46,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:46,994 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:46,994 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:46,994 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:46,994 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:46,994 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:46,998 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,998 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:46,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:46,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:46,998 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:46,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:46,999 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,001 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360886997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360886998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:47,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:47,007 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:47,007 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,007 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:47,008 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,008 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:47,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360887000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:47,008 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:47,008 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,008 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,008 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:47,008 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:47,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/39885dd7027447c6aa05cc854bc871ef 2023-04-01 14:53:47,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360887049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360887054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/158a1470a5f943f5b6f934cca449ec3d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/158a1470a5f943f5b6f934cca449ec3d 2023-04-01 14:53:47,109 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:47,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:47,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360887109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360887109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,114 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/158a1470a5f943f5b6f934cca449ec3d, entries=200, sequenceid=141, filesize=14.1 K 2023-04-01 14:53:47,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360887114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,117 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/5711a6e25e664fb3a5867f3e3faff1a8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5711a6e25e664fb3a5867f3e3faff1a8 2023-04-01 14:53:47,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:47,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:47,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:47,136 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:47,136 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:47,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5711a6e25e664fb3a5867f3e3faff1a8, entries=150, sequenceid=141, filesize=11.8 K 2023-04-01 14:53:47,152 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:47,152 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:47,153 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,153 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/39885dd7027447c6aa05cc854bc871ef as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/39885dd7027447c6aa05cc854bc871ef 2023-04-01 14:53:47,163 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:47,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:47,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:47,163 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,164 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,164 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:47,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:47,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:47,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:47,185 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,185 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:47,185 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:47,185 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:47,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,190 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,190 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,190 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:47,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@545be888[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:47,192 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:47,192 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,194 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:47,194 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:47,194 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:47,194 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,194 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:47,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:47,196 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:47,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:47,199 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:47,199 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:47,199 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:47,199 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:47,199 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:47,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/39885dd7027447c6aa05cc854bc871ef, entries=150, sequenceid=141, filesize=11.8 K 2023-04-01 14:53:47,216 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 8f567672fffd4bc77debf33ecd862db3 in 368ms, sequenceid=141, compaction requested=true 2023-04-01 14:53:47,216 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:47,216 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:47,216 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:53:47,217 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:53:47,216 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:47,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:47,218 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,218 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,218 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:47,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,220 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,221 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50187 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:53:47,221 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/A is initiating minor compaction (all files) 2023-04-01 14:53:47,221 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/A in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,221 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/2ab6282b23f34da488dba7de696a6003, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/4cb570775fb04655b4b3d25cb16e2117, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9641c33393e8466094455e59b19d0434, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/158a1470a5f943f5b6f934cca449ec3d] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=49.0 K 2023-04-01 14:53:47,221 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:53:47,221 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/B is initiating minor compaction (all files) 2023-04-01 14:53:47,221 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/B in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,222 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/34211a328a124e9d99ba7fa32af71aba, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/342e06fa898140018c1f1eac36448297, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6913081a186443f68cc3649a50f4c3b3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5711a6e25e664fb3a5867f3e3faff1a8] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=46.7 K 2023-04-01 14:53:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,224 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 34211a328a124e9d99ba7fa32af71aba, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680360824682 2023-04-01 14:53:47,223 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,223 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 2ab6282b23f34da488dba7de696a6003, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680360824682 2023-04-01 14:53:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,232 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 4cb570775fb04655b4b3d25cb16e2117, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680360824872 2023-04-01 14:53:47,232 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 342e06fa898140018c1f1eac36448297, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680360824872 2023-04-01 14:53:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,233 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 9641c33393e8466094455e59b19d0434, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680360825646 2023-04-01 14:53:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,233 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6913081a186443f68cc3649a50f4c3b3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680360825646 2023-04-01 14:53:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,235 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 5711a6e25e664fb3a5867f3e3faff1a8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680360826834 2023-04-01 14:53:47,235 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,235 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,236 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:47,236 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:47,236 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:47,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:47,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:47,236 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 158a1470a5f943f5b6f934cca449ec3d, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680360826834 2023-04-01 14:53:47,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:47,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:47,236 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,241 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,241 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,241 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,241 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,242 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:47,242 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:47,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:47,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,279 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-01 14:53:47,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:47,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:47,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:47,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:47,280 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:47,280 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:47,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:47,287 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#A#compaction#21 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:47,302 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#B#compaction#22 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:47,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360887318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360887324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360887329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360887330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360887330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,342 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:47,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:47,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:47,350 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/bb36332297f14c51abd836053a6c1e38 2023-04-01 14:53:47,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:47,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:47,368 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:47,368 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:47,369 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:47,369 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:47,370 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,370 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,374 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:47,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:47,374 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,375 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,375 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:47,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:47,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:47,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:47,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:47,391 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:47,391 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:47,394 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,394 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:47,394 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:47,394 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:47,395 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:47,397 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,397 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,398 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,398 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:47,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28991c99[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:47,410 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:47,410 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,413 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:47,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:47,422 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:47,422 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:47,422 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:47,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:47,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:47,436 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:47,436 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:47,436 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:47,436 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:47,436 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:47,441 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,441 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:47,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360887440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,444 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360887440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360887440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360887441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,451 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:47,451 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:47,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:47,452 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,452 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,452 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:47,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:47,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:47,452 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:47,452 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,453 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,453 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,453 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,453 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:47,453 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:47,463 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-01 14:53:47,463 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-01 14:53:47,553 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:47,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:47,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:47,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:47,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:47,560 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:47,560 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:47,561 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:47,561 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:47,562 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,562 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,567 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:47,567 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,568 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,568 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:47,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:47,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:47,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:47,573 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:47,576 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,576 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:47,576 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:47,576 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,576 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:47,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,579 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,579 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:47,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@158c23d2[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:47,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:47,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,587 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:47,587 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:47,587 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:47,588 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,588 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:47,589 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:47,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:47,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:47,593 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:47,594 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:47,594 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:47,594 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:47,594 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:47,596 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,596 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,596 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:47,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,598 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,600 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,603 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,603 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:47,603 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:47,603 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:47,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:47,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:47,603 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:47,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:47,603 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:47,604 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:47,605 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:47,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,605 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,605 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360887636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360887647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360887647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360887649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360887649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,705 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:47,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:47,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:47,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:47,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:47,712 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:47,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:47,717 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:47,717 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:47,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:47,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:47,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:47,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:47,723 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:47,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:47,723 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,724 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:47,724 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:47,724 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:47,734 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,734 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,734 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,735 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:47,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e989c4b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:47,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:47,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,739 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:47,740 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:47,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:47,742 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:47,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:47,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,753 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/74794d9a379e4928a5a4d5566fb82e64 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/74794d9a379e4928a5a4d5566fb82e64 2023-04-01 14:53:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:47,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:47,754 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:47,754 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:47,754 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:47,754 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:47,754 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:47,755 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/70cabfde3a0f4c8e8dc26009cb8dab57 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/70cabfde3a0f4c8e8dc26009cb8dab57 2023-04-01 14:53:47,756 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,756 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,756 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,757 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,760 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:47,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:47,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:47,768 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:47,768 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/B of 8f567672fffd4bc77debf33ecd862db3 into 74794d9a379e4928a5a4d5566fb82e64(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:47,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:47,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:47,770 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:47,768 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,773 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,773 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:47,773 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:47,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:47,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:47,773 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,773 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,773 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,773 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,773 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/B, priority=12, startTime=1680360827216; duration=0sec 2023-04-01 14:53:47,773 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:47,774 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:53:47,776 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/A of 8f567672fffd4bc77debf33ecd862db3 into 70cabfde3a0f4c8e8dc26009cb8dab57(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:47,776 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:47,776 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/A, priority=12, startTime=1680360827216; duration=0sec 2023-04-01 14:53:47,776 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:47,777 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:53:47,777 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/C is initiating minor compaction (all files) 2023-04-01 14:53:47,777 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/C in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,777 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/707ee18b5131436e880a0f0a2428f9f5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/e467c815dace4c0197416ee3458efbed, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/33854bc7845b466b914d098f9b697875, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/39885dd7027447c6aa05cc854bc871ef] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=46.7 K 2023-04-01 14:53:47,778 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 707ee18b5131436e880a0f0a2428f9f5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680360824682 2023-04-01 14:53:47,779 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e467c815dace4c0197416ee3458efbed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680360824872 2023-04-01 14:53:47,780 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 33854bc7845b466b914d098f9b697875, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680360825646 2023-04-01 14:53:47,782 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 39885dd7027447c6aa05cc854bc871ef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680360826834 2023-04-01 14:53:47,797 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#C#compaction#25 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:47,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/15e37978d1ae4066be77dc7846cdc3a3 2023-04-01 14:53:47,873 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:47,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:47,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:47,880 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:47,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:47,881 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:47,881 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:47,882 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,882 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,885 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,885 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:47,885 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,885 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:47,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:47,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:47,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:47,887 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:47,887 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:47,888 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:47,896 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,896 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:47,896 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:47,896 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:47,897 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:47,898 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:47,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@350c3cb8[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:47,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:47,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,904 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,904 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:47,904 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:47,907 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,908 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:47,908 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,908 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:47,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:47,915 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:47,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:47,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:47,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:47,915 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:47,918 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,918 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:47,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,918 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,919 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,921 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:47,925 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:47,925 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:47,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:47,925 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,925 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,925 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:47,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:47,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:47,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:47,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:47,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:47,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:47,929 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:47,929 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:47,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:47,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360887951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360887952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360887955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:47,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:47,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360887957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,029 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:48,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:48,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:48,034 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:48,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:48,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:48,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:48,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,038 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:48,038 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,038 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,038 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:48,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:48,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:48,039 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:48,039 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:48,041 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:48,041 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:48,041 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:48,041 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:48,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,044 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:48,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55119354[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:48,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:48,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,048 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:48,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,049 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:48,050 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:48,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:48,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:48,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:48,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:48,054 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:48,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:48,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:48,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:48,054 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:48,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:48,057 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:48,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,058 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:48,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:48,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:48,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:48,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:48,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,062 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:48,063 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:48,063 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:48,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:48,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:48,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360888143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,163 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:48,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:48,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:48,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:48,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:48,169 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:48,169 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:48,170 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:48,170 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:48,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:48,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:48,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:48,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:48,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:48,174 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:48,174 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:48,175 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:48,175 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:48,175 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:48,175 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:48,175 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:48,175 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:48,175 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:48,175 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:48,175 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:48,175 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:48,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,178 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:48,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:48,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2177b897[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:48,180 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:48,180 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,185 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:48,185 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,186 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:48,185 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:48,186 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:48,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:48,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:48,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:48,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:48,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:48,191 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:48,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:48,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:48,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:48,191 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:48,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:48,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:48,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,197 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:48,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,199 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:48,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:48,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:48,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:48,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:48,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:48,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:48,204 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:48,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,204 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:48,226 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/9fe19587c61a4546b016db370bbaf865 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9fe19587c61a4546b016db370bbaf865 2023-04-01 14:53:48,236 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/C of 8f567672fffd4bc77debf33ecd862db3 into 9fe19587c61a4546b016db370bbaf865(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:48,236 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:48,236 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/C, priority=12, startTime=1680360827217; duration=0sec 2023-04-01 14:53:48,236 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:48,252 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/b1a47fa0a1d2421fa4bf2de0692a29be 2023-04-01 14:53:48,261 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/bb36332297f14c51abd836053a6c1e38 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bb36332297f14c51abd836053a6c1e38 2023-04-01 14:53:48,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bb36332297f14c51abd836053a6c1e38, entries=150, sequenceid=178, filesize=11.8 K 2023-04-01 14:53:48,272 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/15e37978d1ae4066be77dc7846cdc3a3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/15e37978d1ae4066be77dc7846cdc3a3 2023-04-01 14:53:48,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/15e37978d1ae4066be77dc7846cdc3a3, entries=150, sequenceid=178, filesize=11.8 K 2023-04-01 14:53:48,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/b1a47fa0a1d2421fa4bf2de0692a29be as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b1a47fa0a1d2421fa4bf2de0692a29be 2023-04-01 14:53:48,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b1a47fa0a1d2421fa4bf2de0692a29be, entries=150, sequenceid=178, filesize=11.8 K 2023-04-01 14:53:48,302 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for 8f567672fffd4bc77debf33ecd862db3 in 1023ms, sequenceid=178, compaction requested=false 2023-04-01 14:53:48,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:48,304 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:48,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:48,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:48,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:48,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:48,310 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:48,310 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:48,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:48,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:48,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,314 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:48,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:48,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:48,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:48,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:48,316 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:48,316 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:48,317 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:48,317 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:48,317 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-01 14:53:48,318 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:48,318 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:48,318 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:48,318 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:48,318 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:48,318 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:48,342 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/aa7ae60660674470a740fa8d973fe9c5 2023-04-01 14:53:48,379 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/a33c5689d4b44fc8936bec04c7d86b30 2023-04-01 14:53:48,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:48,461 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:48,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360888480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360888481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360828476, max timestamp=1680360828482Num uniques 859; 2023-04-01 14:53:48,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:53:48,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680360828472, max timestamp=1680360828482Num uniques 859; 2023-04-01 14:53:48,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-01 14:53:48,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680360828472, max timestamp=1680360828482Num uniques 859; 2023-04-01 14:53:48,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-01 14:53:48,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360888540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360888542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360888588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360888588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360888645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360888650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360888794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360888795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,818 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/5d95db6bf513463f85c658f3bea4a907 2023-04-01 14:53:48,828 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/aa7ae60660674470a740fa8d973fe9c5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/aa7ae60660674470a740fa8d973fe9c5 2023-04-01 14:53:48,838 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/aa7ae60660674470a740fa8d973fe9c5, entries=150, sequenceid=192, filesize=11.8 K 2023-04-01 14:53:48,842 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/a33c5689d4b44fc8936bec04c7d86b30 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/a33c5689d4b44fc8936bec04c7d86b30 2023-04-01 14:53:48,853 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/a33c5689d4b44fc8936bec04c7d86b30, entries=150, sequenceid=192, filesize=11.8 K 2023-04-01 14:53:48,857 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/5d95db6bf513463f85c658f3bea4a907 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5d95db6bf513463f85c658f3bea4a907 2023-04-01 14:53:48,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360888851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:48,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360888856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,880 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5d95db6bf513463f85c658f3bea4a907, entries=150, sequenceid=192, filesize=11.8 K 2023-04-01 14:53:48,889 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8f567672fffd4bc77debf33ecd862db3 in 571ms, sequenceid=192, compaction requested=true 2023-04-01 14:53:48,889 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:48,889 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:48,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:48,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:48,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:48,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:48,895 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,895 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:48,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:48,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b771236[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:48,899 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:48,899 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,900 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,900 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,900 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:48,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:48,901 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:48,901 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:48,903 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:48,903 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,903 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:48,903 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:48,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:48,909 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:48,909 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:48,909 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:48,909 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:48,909 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:48,912 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,912 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:48,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,912 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:48,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:48,914 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,916 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:48,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:48,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,921 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:48,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:48,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:48,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:48,932 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:48,933 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:48,933 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:48,932 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,933 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:48,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:48,933 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,933 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:48,933 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:48,933 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,933 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:48,933 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:48,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:49,033 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:49,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:49,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:49,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:49,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:49,039 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:49,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:49,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:49,041 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,041 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,042 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:49,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:49,043 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,043 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:49,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:49,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:49,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:49,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:49,049 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,049 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:49,049 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-01 14:53:49,049 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:49,049 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:49,049 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:49,050 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:49,050 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:49,050 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:49,074 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/52e51d8b5e89470e87a50be53f622d24 2023-04-01 14:53:49,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:49,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:49,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360889117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,123 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/40adb6ee151f42a2bba2cbb7b47225d9 2023-04-01 14:53:49,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360889122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,148 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/cc7f7adf311c4ce8adca0593fcd48ea4 2023-04-01 14:53:49,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360889151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,156 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/52e51d8b5e89470e87a50be53f622d24 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/52e51d8b5e89470e87a50be53f622d24 2023-04-01 14:53:49,170 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/52e51d8b5e89470e87a50be53f622d24, entries=150, sequenceid=226, filesize=11.8 K 2023-04-01 14:53:49,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360889170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/40adb6ee151f42a2bba2cbb7b47225d9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/40adb6ee151f42a2bba2cbb7b47225d9 2023-04-01 14:53:49,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360889170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,183 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/40adb6ee151f42a2bba2cbb7b47225d9, entries=150, sequenceid=226, filesize=11.8 K 2023-04-01 14:53:49,186 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/cc7f7adf311c4ce8adca0593fcd48ea4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cc7f7adf311c4ce8adca0593fcd48ea4 2023-04-01 14:53:49,194 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cc7f7adf311c4ce8adca0593fcd48ea4, entries=150, sequenceid=226, filesize=11.8 K 2023-04-01 14:53:49,197 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 8f567672fffd4bc77debf33ecd862db3 in 148ms, sequenceid=226, compaction requested=true 2023-04-01 14:53:49,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:49,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,203 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,203 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:49,205 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fca1c1c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:49,205 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,206 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,206 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:49,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:49,206 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:49,207 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:49,209 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,209 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,209 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:49,209 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:49,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:49,215 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:49,215 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:49,215 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:49,215 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:49,215 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:49,217 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,217 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:49,217 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:49,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,222 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:49,225 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 14:53:49,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:49,226 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:49,226 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:49,226 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:49,226 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:49,226 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:49,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:49,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:49,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:49,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:49,227 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,227 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:49,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:49,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:49,227 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,229 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,229 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,229 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:49,229 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:49,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:49,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/7a87792459884c50b01956c9e052ffe2 2023-04-01 14:53:49,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360889276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360889280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,329 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:49,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:49,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:49,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:49,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:49,334 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:49,334 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,335 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:49,335 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:49,335 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,336 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:49,337 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:49,337 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,337 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:49,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:49,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:49,341 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:49,341 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,342 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:49,342 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:49,342 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:49,342 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,342 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:49,342 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:49,342 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:49,344 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,344 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:49,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5698b582[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:49,350 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:49,350 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,352 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,352 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:49,353 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,353 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:49,353 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:49,353 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:49,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,356 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:49,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,356 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:49,356 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:49,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:49,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:49,360 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:49,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:49,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:49,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:49,360 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:49,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,366 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:49,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:49,366 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,368 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,371 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:49,371 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:49,371 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:49,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:49,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:49,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:49,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,373 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:49,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:49,373 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:49,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:49,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360889382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360889384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,473 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:49,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:49,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:49,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:49,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:49,479 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:49,479 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:49,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:49,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:49,482 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:49,482 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,482 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:49,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:49,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:49,484 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:49,484 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,485 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:49,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:49,488 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,488 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:49,488 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:49,488 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:49,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:49,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:49,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:49,491 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,491 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:49,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a370cc6[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:49,494 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:49,494 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,495 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,495 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:49,495 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:49,495 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:49,495 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,495 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:49,496 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:49,496 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,496 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,497 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:49,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:49,500 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:49,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:49,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:49,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:49,501 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:49,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:49,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,503 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,505 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,510 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,510 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,510 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:49,510 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,510 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:49,510 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,510 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,510 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,510 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:49,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:49,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:49,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:49,510 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:49,511 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:49,512 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:49,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:49,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360889587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360889588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,612 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:49,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:49,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:49,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:49,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:49,621 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:49,621 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,622 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:49,623 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:49,623 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,623 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,624 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:49,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:49,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,625 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,625 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:49,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:49,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:49,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:49,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,629 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:49,630 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,630 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:49,630 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:49,630 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:49,632 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,632 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,633 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,633 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:49,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29b24114[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:49,636 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:49,636 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,639 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:49,640 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,640 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:49,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,642 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:49,642 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:49,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:49,647 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:49,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:49,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:49,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:49,647 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:49,653 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,653 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,654 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:49,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,654 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,656 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,659 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,659 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,659 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,659 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,660 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,660 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:49,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:49,660 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:49,660 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:49,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:49,660 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:49,660 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:49,660 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:49,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:49,660 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/c3d7e9c3fa5a46fa8952eb3ec14b1e2d 2023-04-01 14:53:49,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360889694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360889694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,739 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/8ecdc287ba7b40c3859a979ff13d3cb5 2023-04-01 14:53:49,748 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/7a87792459884c50b01956c9e052ffe2 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a87792459884c50b01956c9e052ffe2 2023-04-01 14:53:49,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a87792459884c50b01956c9e052ffe2, entries=200, sequenceid=238, filesize=14.1 K 2023-04-01 14:53:49,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/c3d7e9c3fa5a46fa8952eb3ec14b1e2d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c3d7e9c3fa5a46fa8952eb3ec14b1e2d 2023-04-01 14:53:49,760 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:49,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:49,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:49,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c3d7e9c3fa5a46fa8952eb3ec14b1e2d, entries=150, sequenceid=238, filesize=11.8 K 2023-04-01 14:53:49,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/8ecdc287ba7b40c3859a979ff13d3cb5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/8ecdc287ba7b40c3859a979ff13d3cb5 2023-04-01 14:53:49,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:49,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:49,770 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:49,770 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,770 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:49,770 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:49,772 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,772 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,775 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,775 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:49,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:49,775 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,775 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:49,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:49,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:49,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:49,784 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,784 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:49,784 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:49,784 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:49,792 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:49,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/8ecdc287ba7b40c3859a979ff13d3cb5, entries=150, sequenceid=238, filesize=11.8 K 2023-04-01 14:53:49,798 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8f567672fffd4bc77debf33ecd862db3 in 573ms, sequenceid=238, compaction requested=true 2023-04-01 14:53:49,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:49,798 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:49,798 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:53:49,798 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:53:49,798 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:49,798 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:49,800 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,800 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,802 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62810 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:53:49,802 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/A is initiating minor compaction (all files) 2023-04-01 14:53:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,802 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/A in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,803 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/70cabfde3a0f4c8e8dc26009cb8dab57, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bb36332297f14c51abd836053a6c1e38, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/aa7ae60660674470a740fa8d973fe9c5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/52e51d8b5e89470e87a50be53f622d24, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a87792459884c50b01956c9e052ffe2] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=61.3 K 2023-04-01 14:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@764792c2[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:49,803 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60420 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:53:49,803 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:49,805 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,805 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/B is initiating minor compaction (all files) 2023-04-01 14:53:49,805 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/B in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,805 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/74794d9a379e4928a5a4d5566fb82e64, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/15e37978d1ae4066be77dc7846cdc3a3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/a33c5689d4b44fc8936bec04c7d86b30, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/40adb6ee151f42a2bba2cbb7b47225d9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c3d7e9c3fa5a46fa8952eb3ec14b1e2d] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=59.0 K 2023-04-01 14:53:49,806 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 74794d9a379e4928a5a4d5566fb82e64, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680360826834 2023-04-01 14:53:49,806 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:49,817 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,817 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 15e37978d1ae4066be77dc7846cdc3a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680360826996 2023-04-01 14:53:49,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:49,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:49,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:49,822 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 70cabfde3a0f4c8e8dc26009cb8dab57, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680360826834 2023-04-01 14:53:49,822 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,822 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a33c5689d4b44fc8936bec04c7d86b30, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680360827290 2023-04-01 14:53:49,822 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:49,823 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 40adb6ee151f42a2bba2cbb7b47225d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680360828532 2023-04-01 14:53:49,823 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting bb36332297f14c51abd836053a6c1e38, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680360826996 2023-04-01 14:53:49,824 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c3d7e9c3fa5a46fa8952eb3ec14b1e2d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680360829114 2023-04-01 14:53:49,824 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting aa7ae60660674470a740fa8d973fe9c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680360827290 2023-04-01 14:53:49,825 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,825 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 52e51d8b5e89470e87a50be53f622d24, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680360828532 2023-04-01 14:53:49,825 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:49,826 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:49,826 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,828 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 7a87792459884c50b01956c9e052ffe2, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680360829109 2023-04-01 14:53:49,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:49,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:49,834 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:49,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:49,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:49,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:49,834 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:49,846 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#A#compaction#36 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:49,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:49,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:49,848 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:49,850 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:49,857 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:49,857 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:49,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:49,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:49,858 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#B#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:49,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,860 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:49,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:49,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:49,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,860 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:49,860 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:49,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:49,883 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/0f78789ccabf4ab1a7afb236e11f28bc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/0f78789ccabf4ab1a7afb236e11f28bc 2023-04-01 14:53:49,895 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/A of 8f567672fffd4bc77debf33ecd862db3 into 0f78789ccabf4ab1a7afb236e11f28bc(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:49,895 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:49,895 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/A, priority=11, startTime=1680360829798; duration=0sec 2023-04-01 14:53:49,895 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:49,896 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:53:49,899 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60420 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:53:49,899 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/C is initiating minor compaction (all files) 2023-04-01 14:53:49,899 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/C in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,900 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9fe19587c61a4546b016db370bbaf865, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b1a47fa0a1d2421fa4bf2de0692a29be, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5d95db6bf513463f85c658f3bea4a907, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cc7f7adf311c4ce8adca0593fcd48ea4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/8ecdc287ba7b40c3859a979ff13d3cb5] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=59.0 K 2023-04-01 14:53:49,900 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 9fe19587c61a4546b016db370bbaf865, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1680360826834 2023-04-01 14:53:49,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:49,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-01 14:53:49,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:49,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:49,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:49,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:49,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:49,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:49,904 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting b1a47fa0a1d2421fa4bf2de0692a29be, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680360826996 2023-04-01 14:53:49,905 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 5d95db6bf513463f85c658f3bea4a907, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680360827290 2023-04-01 14:53:49,906 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting cc7f7adf311c4ce8adca0593fcd48ea4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680360828532 2023-04-01 14:53:49,907 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 8ecdc287ba7b40c3859a979ff13d3cb5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680360829114 2023-04-01 14:53:49,927 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/e350ed8b7b4c4b598f299eb9a395adbe as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e350ed8b7b4c4b598f299eb9a395adbe 2023-04-01 14:53:49,930 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#C#compaction#39 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:49,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360889937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:49,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360889937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,947 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/B of 8f567672fffd4bc77debf33ecd862db3 into e350ed8b7b4c4b598f299eb9a395adbe(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:49,947 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:49,947 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/B, priority=11, startTime=1680360829798; duration=0sec 2023-04-01 14:53:49,947 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:49,960 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:49,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:49,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:49,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:49,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:49,987 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:49,987 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,988 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:49,988 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:49,989 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,989 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:49,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:49,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:49,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:49,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:49,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:49,995 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,995 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:49,995 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:49,995 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:49,996 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:49,996 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:49,996 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:49,996 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:49,996 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:49,999 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:49,999 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,000 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/b9415ec1edf041a18fdf98e847694f76 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b9415ec1edf041a18fdf98e847694f76 2023-04-01 14:53:50,000 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,000 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:50,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65e01d2d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:50,003 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:50,003 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,004 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,006 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:50,006 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:50,006 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:50,006 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,006 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:50,008 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:50,008 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,008 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:50,008 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,011 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/C of 8f567672fffd4bc77debf33ecd862db3 into b9415ec1edf041a18fdf98e847694f76(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:50,011 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,011 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/C, priority=11, startTime=1680360829798; duration=0sec 2023-04-01 14:53:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,011 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:50,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:50,013 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:50,013 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:50,013 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:50,013 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:50,013 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:50,015 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,015 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,015 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,017 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,019 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,025 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:50,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:50,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:50,026 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,027 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:50,027 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:50,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360890042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360890043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,127 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:50,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:50,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:50,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:50,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:50,133 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:50,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:50,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:50,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,135 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,136 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,136 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:50,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:50,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:50,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:50,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:50,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:50,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:50,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17437c6b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:50,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:50,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,151 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:50,152 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,152 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:50,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:50,158 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:50,158 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:50,158 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:50,158 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:50,158 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:50,161 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,161 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,161 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,165 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:50,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:50,168 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,168 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:50,168 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,170 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,170 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,170 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,170 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,170 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,170 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360890247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360890247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,268 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:50,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:50,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:50,274 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:50,274 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,274 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:50,274 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:50,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,276 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,277 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,277 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:50,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:50,282 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,282 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:50,282 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:50,282 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:50,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c423576[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:50,291 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:50,291 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,293 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:50,293 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,293 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:50,295 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:50,298 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:50,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:50,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:50,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:50,298 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:50,300 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,300 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,301 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,305 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,314 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,314 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:50,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:50,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,314 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:50,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,315 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:50,315 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:50,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/debaf99cd651438a93aac38bcd809beb 2023-04-01 14:53:50,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/c4b4224dbdb84f4eaca648dfc91e188a 2023-04-01 14:53:50,415 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:50,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:50,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:50,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:50,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:50,424 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:50,424 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,424 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:50,424 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:50,425 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,425 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,426 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,426 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,427 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:50,427 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,427 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/695bb9ee66524f7a8da3d89071909d49 2023-04-01 14:53:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:50,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:50,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:50,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:50,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:50,434 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,435 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,435 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@665963be[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:50,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:50,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/debaf99cd651438a93aac38bcd809beb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/debaf99cd651438a93aac38bcd809beb 2023-04-01 14:53:50,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:50,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:50,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:50,443 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,443 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:50,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:50,448 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:50,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:50,455 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:50,455 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:50,455 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:50,455 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:50,455 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:50,457 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,457 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/debaf99cd651438a93aac38bcd809beb, entries=200, sequenceid=260, filesize=14.2 K 2023-04-01 14:53:50,457 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,459 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/c4b4224dbdb84f4eaca648dfc91e188a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4b4224dbdb84f4eaca648dfc91e188a 2023-04-01 14:53:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,460 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:50,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:50,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:50,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,464 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:50,464 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:50,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4b4224dbdb84f4eaca648dfc91e188a, entries=150, sequenceid=260, filesize=11.8 K 2023-04-01 14:53:50,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/695bb9ee66524f7a8da3d89071909d49 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/695bb9ee66524f7a8da3d89071909d49 2023-04-01 14:53:50,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/695bb9ee66524f7a8da3d89071909d49, entries=150, sequenceid=260, filesize=11.8 K 2023-04-01 14:53:50,499 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 8f567672fffd4bc77debf33ecd862db3 in 598ms, sequenceid=260, compaction requested=false 2023-04-01 14:53:50,499 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:50,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:50,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 14:53:50,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:50,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:50,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:50,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:50,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:50,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:50,564 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:50,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:50,567 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/b1f5c00b84854934a2c1a8db83ca0bbc 2023-04-01 14:53:50,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:50,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:50,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:50,569 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:50,569 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,569 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:50,569 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:50,570 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,570 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:50,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:50,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:50,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:50,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:50,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:50,579 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,580 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:50,580 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:50,580 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:50,584 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,584 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,584 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,584 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@335032d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:50,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:50,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/5fd703e9f9db479db7715eabaf23c7ee 2023-04-01 14:53:50,589 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:50,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:50,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:50,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:50,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:50,598 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:50,598 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:50,598 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:50,598 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:50,598 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:50,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,601 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,609 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,609 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,609 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,609 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:50,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:50,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,609 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:50,609 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/2a11458b76ba4bf6bfcf61ebc13f5598 2023-04-01 14:53:50,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/b1f5c00b84854934a2c1a8db83ca0bbc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b1f5c00b84854934a2c1a8db83ca0bbc 2023-04-01 14:53:50,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360890616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360890619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b1f5c00b84854934a2c1a8db83ca0bbc, entries=150, sequenceid=274, filesize=11.9 K 2023-04-01 14:53:50,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/5fd703e9f9db479db7715eabaf23c7ee as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5fd703e9f9db479db7715eabaf23c7ee 2023-04-01 14:53:50,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5fd703e9f9db479db7715eabaf23c7ee, entries=100, sequenceid=274, filesize=9.5 K 2023-04-01 14:53:50,646 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/2a11458b76ba4bf6bfcf61ebc13f5598 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/2a11458b76ba4bf6bfcf61ebc13f5598 2023-04-01 14:53:50,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/2a11458b76ba4bf6bfcf61ebc13f5598, entries=100, sequenceid=274, filesize=9.5 K 2023-04-01 14:53:50,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8f567672fffd4bc77debf33ecd862db3 in 100ms, sequenceid=274, compaction requested=true 2023-04-01 14:53:50,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:50,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:50,655 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:53:50,655 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:53:50,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:50,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:50,657 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39154 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:53:50,657 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/A is initiating minor compaction (all files) 2023-04-01 14:53:50,657 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:53:50,657 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/A in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,657 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/B is initiating minor compaction (all files) 2023-04-01 14:53:50,657 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/0f78789ccabf4ab1a7afb236e11f28bc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/debaf99cd651438a93aac38bcd809beb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b1f5c00b84854934a2c1a8db83ca0bbc] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=38.2 K 2023-04-01 14:53:50,657 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/B in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,657 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e350ed8b7b4c4b598f299eb9a395adbe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4b4224dbdb84f4eaca648dfc91e188a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5fd703e9f9db479db7715eabaf23c7ee] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=33.5 K 2023-04-01 14:53:50,657 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 0f78789ccabf4ab1a7afb236e11f28bc, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680360829114 2023-04-01 14:53:50,658 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e350ed8b7b4c4b598f299eb9a395adbe, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680360829114 2023-04-01 14:53:50,662 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting debaf99cd651438a93aac38bcd809beb, keycount=200, bloomtype=ROW, size=14.2 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680360829271 2023-04-01 14:53:50,662 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c4b4224dbdb84f4eaca648dfc91e188a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680360829271 2023-04-01 14:53:50,663 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting b1f5c00b84854934a2c1a8db83ca0bbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680360829923 2023-04-01 14:53:50,663 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 5fd703e9f9db479db7715eabaf23c7ee, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680360829923 2023-04-01 14:53:50,675 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#A#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:50,676 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#B#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:50,710 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:50,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:50,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-04-01 14:53:50,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:50,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:50,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:50,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:50,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:50,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:50,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:50,717 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/0626ae85cb8b42cb96d8dbf5075ef3d4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/0626ae85cb8b42cb96d8dbf5075ef3d4 2023-04-01 14:53:50,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:50,732 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/B of 8f567672fffd4bc77debf33ecd862db3 into 0626ae85cb8b42cb96d8dbf5075ef3d4(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:50,732 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:50,732 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/B, priority=13, startTime=1680360830655; duration=0sec 2023-04-01 14:53:50,732 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:50,732 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:53:50,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:50,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:50,734 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:50,734 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,734 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:50,735 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:50,735 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,735 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:53:50,735 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/C is initiating minor compaction (all files) 2023-04-01 14:53:50,735 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,735 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/C in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,735 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b9415ec1edf041a18fdf98e847694f76, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/695bb9ee66524f7a8da3d89071909d49, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/2a11458b76ba4bf6bfcf61ebc13f5598] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=33.5 K 2023-04-01 14:53:50,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360890732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,736 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b9415ec1edf041a18fdf98e847694f76, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680360829114 2023-04-01 14:53:50,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,737 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360890735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,737 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360890737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360890737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,750 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 695bb9ee66524f7a8da3d89071909d49, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680360829271 2023-04-01 14:53:50,751 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2a11458b76ba4bf6bfcf61ebc13f5598, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680360829923 2023-04-01 14:53:50,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:50,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:50,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:50,758 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:50,758 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,758 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:50,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/7a1540b62abd43549b542e01dbf1d61e 2023-04-01 14:53:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:50,770 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,770 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:50,770 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:50,770 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,770 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:50,771 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#C#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,772 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f199704[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:50,775 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:50,775 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:50,778 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,778 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:50,781 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:50,781 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,781 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:50,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,794 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/58cb89e666f8437191a8ce43c81b97cc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/58cb89e666f8437191a8ce43c81b97cc 2023-04-01 14:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:50,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:50,795 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:50,795 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:50,795 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:50,795 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:50,795 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:50,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,798 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,799 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,804 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:50,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:50,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:50,805 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:50,805 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:50,806 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/C of 8f567672fffd4bc77debf33ecd862db3 into 58cb89e666f8437191a8ce43c81b97cc(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:50,806 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:50,806 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/C, priority=13, startTime=1680360830655; duration=0sec 2023-04-01 14:53:50,806 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:50,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/be7ce9140753472693346e70dd95b94c 2023-04-01 14:53:50,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360890838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360890851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360890852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:50,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360890852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,905 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:50,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:50,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:50,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:50,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:50,911 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:50,911 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,911 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:50,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:50,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,913 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:50,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:50,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:50,920 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,921 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:50,921 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:50,921 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:50,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:50,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:50,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:50,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:50,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32cdfa14[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:50,930 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:50,930 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,932 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:50,932 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,932 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:50,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:50,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:50,938 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:50,938 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:50,938 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:50,938 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:50,938 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:50,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:50,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,943 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:50,946 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,951 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:50,951 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:50,951 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:50,951 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,951 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:50,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:50,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:50,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:50,951 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:50,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:50,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:50,952 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:50,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:50,952 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:51,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360891043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,052 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:51,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:51,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360891055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:51,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360891057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360891058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:51,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:51,060 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:51,060 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,060 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:51,060 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:51,060 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,060 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,062 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,063 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,063 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:51,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:51,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:51,070 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,070 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:51,070 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:51,070 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:51,073 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa42b60[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:51,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:51,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,080 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,080 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:51,080 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:51,080 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:51,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:51,083 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,086 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:51,086 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,086 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:51,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:51,089 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:51,090 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:51,090 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:51,090 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:51,090 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:51,092 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,092 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,092 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,093 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,100 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:51,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:51,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,104 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,104 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:51,104 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:51,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:51,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,115 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/aa14893980714543a540926875608171 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/aa14893980714543a540926875608171 2023-04-01 14:53:51,125 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/A of 8f567672fffd4bc77debf33ecd862db3 into aa14893980714543a540926875608171(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:51,125 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:51,125 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/A, priority=13, startTime=1680360830655; duration=0sec 2023-04-01 14:53:51,125 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:51,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360891167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,173 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:53:51,204 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:51,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:51,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:51,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:51,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:51,211 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:51,211 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,211 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:51,211 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:51,212 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,212 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,213 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,213 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,217 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,217 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:51,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:51,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:51,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:51,219 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:51,219 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,219 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:51,219 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:51,219 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:51,219 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,219 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:51,219 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:51,219 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,220 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:51,220 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:51,220 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,220 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:51,220 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:51,223 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,223 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,224 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:51,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34a36fd2[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:51,225 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:51,225 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:51,227 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:51,228 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,228 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:51,228 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:51,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:51,232 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:51,232 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:51,232 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:51,232 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:51,232 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:51,235 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,235 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,235 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,235 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,237 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:51,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,240 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:51,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,241 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,241 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,241 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,241 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:51,241 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:51,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,252 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/20acbd7f759c497da886476a17c4df19 2023-04-01 14:53:51,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/7a1540b62abd43549b542e01dbf1d61e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a1540b62abd43549b542e01dbf1d61e 2023-04-01 14:53:51,265 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a1540b62abd43549b542e01dbf1d61e, entries=150, sequenceid=296, filesize=11.9 K 2023-04-01 14:53:51,266 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/be7ce9140753472693346e70dd95b94c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/be7ce9140753472693346e70dd95b94c 2023-04-01 14:53:51,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/be7ce9140753472693346e70dd95b94c, entries=150, sequenceid=296, filesize=11.9 K 2023-04-01 14:53:51,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/20acbd7f759c497da886476a17c4df19 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20acbd7f759c497da886476a17c4df19 2023-04-01 14:53:51,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20acbd7f759c497da886476a17c4df19, entries=150, sequenceid=296, filesize=11.9 K 2023-04-01 14:53:51,282 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=53.67 KB/54960 for 8f567672fffd4bc77debf33ecd862db3 in 569ms, sequenceid=296, compaction requested=false 2023-04-01 14:53:51,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:51,341 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:51,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:51,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:51,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:51,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:51,346 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:51,346 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,346 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:51,346 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:51,346 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,348 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,348 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,348 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,348 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:51,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:51,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:51,350 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 14:53:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:51,350 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:51,350 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,350 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:51,350 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:51,350 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:51,350 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:51,350 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,350 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:51,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:51,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:51,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:51,353 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,353 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,353 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,353 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77c17ff5[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:51,356 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:51,356 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,361 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:51,361 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:51,361 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:51,361 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,361 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:51,362 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/c2fa08a8410942399fef5331f0af4271 2023-04-01 14:53:51,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:51,374 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:51,374 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:51,374 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:51,374 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:51,375 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:51,377 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,377 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,377 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680360831357, max timestamp=1680360831377Num uniques 859; 2023-04-01 14:53:51,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-01 14:53:51,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360831370, max timestamp=1680360831377Num uniques 859; 2023-04-01 14:53:51,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:53:51,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360831370, max timestamp=1680360831377Num uniques 859; 2023-04-01 14:53:51,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:53:51,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,385 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,387 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,390 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,390 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,390 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,390 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,390 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,390 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,390 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,390 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:51,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:51,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,390 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,390 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:51,390 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:51,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360891401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360891401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360891403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360891408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/dbaf61f4e3b9431eb0aed46b2fef3713 2023-04-01 14:53:51,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/574052d55a6846df816207b8f429706a 2023-04-01 14:53:51,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/c2fa08a8410942399fef5331f0af4271 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c2fa08a8410942399fef5331f0af4271 2023-04-01 14:53:51,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c2fa08a8410942399fef5331f0af4271, entries=150, sequenceid=311, filesize=11.9 K 2023-04-01 14:53:51,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/dbaf61f4e3b9431eb0aed46b2fef3713 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dbaf61f4e3b9431eb0aed46b2fef3713 2023-04-01 14:53:51,490 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:51,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:51,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:51,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:51,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:51,497 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:51,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:51,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:51,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,497 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dbaf61f4e3b9431eb0aed46b2fef3713, entries=150, sequenceid=311, filesize=11.9 K 2023-04-01 14:53:51,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/574052d55a6846df816207b8f429706a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/574052d55a6846df816207b8f429706a 2023-04-01 14:53:51,501 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:51,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:51,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:51,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:51,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:51,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:51,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:51,504 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,504 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:51,504 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:51,504 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:51,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:51,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/574052d55a6846df816207b8f429706a, entries=150, sequenceid=311, filesize=11.9 K 2023-04-01 14:53:51,506 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=123.00 KB/125950 for 8f567672fffd4bc77debf33ecd862db3 in 156ms, sequenceid=311, compaction requested=true 2023-04-01 14:53:51,506 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:51,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:51,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 14:53:51,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-01 14:53:51,506 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:53:51,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,507 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,507 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,507 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,508 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:51,508 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:53:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,508 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/A is initiating minor compaction (all files) 2023-04-01 14:53:51,508 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/A in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,508 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/aa14893980714543a540926875608171, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a1540b62abd43549b542e01dbf1d61e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c2fa08a8410942399fef5331f0af4271] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=36.2 K 2023-04-01 14:53:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,510 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting aa14893980714543a540926875608171, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680360829271 2023-04-01 14:53:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:51,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a812643[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:51,514 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:51,514 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,514 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:53:51,514 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/C is initiating minor compaction (all files) 2023-04-01 14:53:51,514 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/C in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,514 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/58cb89e666f8437191a8ce43c81b97cc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20acbd7f759c497da886476a17c4df19, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/574052d55a6846df816207b8f429706a] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=36.2 K 2023-04-01 14:53:51,515 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 7a1540b62abd43549b542e01dbf1d61e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680360830609 2023-04-01 14:53:51,515 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 58cb89e666f8437191a8ce43c81b97cc, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680360829271 2023-04-01 14:53:51,515 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,515 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c2fa08a8410942399fef5331f0af4271, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680360830717 2023-04-01 14:53:51,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,515 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:51,515 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:51,516 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,516 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:51,516 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 20acbd7f759c497da886476a17c4df19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680360830609 2023-04-01 14:53:51,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:51,517 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=129.71 KB heapSize=365.45 KB 2023-04-01 14:53:51,517 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 574052d55a6846df816207b8f429706a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680360830717 2023-04-01 14:53:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:51,517 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:51,518 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:51,518 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:51,518 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:51,518 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:51,518 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:51,518 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:51,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:51,528 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:51,528 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:51,528 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:51,528 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:51,529 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:51,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360891527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360891528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,534 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,534 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,534 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,535 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360891531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360891531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,541 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:51,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:51,542 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:51,542 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:51,547 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#C#compaction#55 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:51,551 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#A#compaction#56 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:51,580 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/01b9266a0a644957bf05a6d0b4092dc7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/01b9266a0a644957bf05a6d0b4092dc7 2023-04-01 14:53:51,587 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/A of 8f567672fffd4bc77debf33ecd862db3 into 01b9266a0a644957bf05a6d0b4092dc7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:51,588 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:51,588 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/A, priority=13, startTime=1680360831506; duration=0sec 2023-04-01 14:53:51,589 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:51,589 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:53:51,590 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:53:51,590 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/B is initiating minor compaction (all files) 2023-04-01 14:53:51,590 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/B in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,590 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/0626ae85cb8b42cb96d8dbf5075ef3d4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/be7ce9140753472693346e70dd95b94c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dbaf61f4e3b9431eb0aed46b2fef3713] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=36.2 K 2023-04-01 14:53:51,591 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 0626ae85cb8b42cb96d8dbf5075ef3d4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680360829271 2023-04-01 14:53:51,592 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting be7ce9140753472693346e70dd95b94c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680360830609 2023-04-01 14:53:51,593 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting dbaf61f4e3b9431eb0aed46b2fef3713, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680360830717 2023-04-01 14:53:51,601 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#B#compaction#57 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:51,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360891634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360891634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51650 deadline: 1680360891637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,642 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:51,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:51,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:51,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:51,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360891644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:51,647 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:51,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:51,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:51,648 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,648 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,649 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,649 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,650 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,650 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:51,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:51,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:51,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:51,652 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:51,652 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:51,652 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,652 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:51,653 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:51,653 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:51,654 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,654 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:51,658 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:51,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1440e758[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:51,658 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,660 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:51,661 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,661 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:51,662 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,662 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:51,662 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,662 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:51,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:51,666 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:51,666 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:51,666 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:51,666 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:51,666 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:51,668 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,668 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,668 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,670 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,673 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,673 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,673 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,673 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:51,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:51,673 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,673 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,673 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,673 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,673 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,673 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:51,673 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,674 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:51,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,774 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:51,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:51,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:51,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:51,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:51,779 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:51,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:51,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:51,780 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,780 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,781 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,781 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,781 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,782 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:51,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:51,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:51,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:51,784 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,784 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:51,784 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:51,784 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:51,788 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,789 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,789 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bb2d7fd[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:51,796 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:51,796 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,798 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:51,798 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:51,798 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:51,798 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,799 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:51,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,804 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:51,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:51,821 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:51,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:51,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:51,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:51,821 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:51,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,825 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,828 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,835 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,840 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,840 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,840 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,840 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,840 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,840 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,841 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,841 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,841 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,841 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:51,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:51,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:51,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51666 deadline: 1680360891838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,842 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:51,842 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:51,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360891839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360891843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:51,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51728 deadline: 1680360891849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,942 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:51,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:51,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:51,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:51,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:51,947 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:51,947 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,947 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:51,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:51,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,950 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:51,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:51,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:51,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:51,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:51,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:51,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:51,958 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:51,958 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:51,958 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:51,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:51,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:51,969 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,969 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:51,969 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:51,969 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/cc8bfc12d6064eb58a865cf1061ae2cd 2023-04-01 14:53:51,969 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:51,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:51,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:51,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:51,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,972 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:51,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477029a9[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:51,975 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:51,975 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,976 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:51,977 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:51,977 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,977 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:51,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:51,984 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:51,984 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:51,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:51,986 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/c61e51d14246431a82a481f86edc4f8e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/c61e51d14246431a82a481f86edc4f8e 2023-04-01 14:53:51,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:51,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:51,994 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:51,994 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:51,994 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:51,994 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:51,994 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:51,996 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,996 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:51,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,996 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:51,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:51,998 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:52,000 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/C of 8f567672fffd4bc77debf33ecd862db3 into c61e51d14246431a82a481f86edc4f8e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:52,000 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:52,000 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/C, priority=13, startTime=1680360831506; duration=0sec 2023-04-01 14:53:52,000 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:52,000 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,010 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,010 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,010 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,010 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,010 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:52,010 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:52,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:52,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:52,010 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:52,010 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,011 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,011 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:52,011 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:52,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:52,011 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:52,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:52,019 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/c4a5c024c2884fa7a47fcb86c0e99d06 2023-04-01 14:53:52,024 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/429ccab98ddc48abbff9ae5c828e5af8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/429ccab98ddc48abbff9ae5c828e5af8 2023-04-01 14:53:52,037 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/B of 8f567672fffd4bc77debf33ecd862db3 into 429ccab98ddc48abbff9ae5c828e5af8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:52,037 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:52,037 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/B, priority=13, startTime=1680360831506; duration=0sec 2023-04-01 14:53:52,037 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:52,111 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:52,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:52,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:52,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:52,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:52,128 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:52,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:52,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:52,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:52,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,130 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:52,130 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:52,130 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,130 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:52,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:52,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:52,132 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:52,132 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:52,132 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:52,132 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:52,134 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,134 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:52,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:52,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6460ac3a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:52,137 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:52,137 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,138 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,138 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:52,138 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:52,138 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:52,139 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,139 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:52,140 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,140 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:52,140 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,140 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:52,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:52,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:52,144 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:52,144 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:52,144 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:52,144 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:52,144 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:52,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:52,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:52,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:52,147 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:52,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360892145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360892146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360892150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360892155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:52,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:52,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:52,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:52,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:52,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:52,163 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:52,163 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:52,163 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:52,163 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,163 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,163 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,163 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:52,263 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:52,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:52,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:52,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:52,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:52,275 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:52,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:52,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:52,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:52,276 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,276 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,277 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,277 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:52,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:52,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:52,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:52,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:52,283 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:52,283 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:52,283 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:52,283 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:52,283 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:52,284 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:52,284 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:52,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,285 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:52,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:52,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:52,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7789edc9[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:52,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:52,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:52,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:52,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:52,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:52,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:52,291 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:52,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:52,295 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:52,295 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:52,295 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:52,295 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:52,295 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:52,296 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,296 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:52,296 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:52,297 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:52,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:52,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:52,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:52,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:52,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:52,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:52,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,304 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:52,304 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:52,404 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:52,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:52,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:52,414 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:52,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:52,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:52,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:52,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:52,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:52,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:52,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:52,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:52,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:52,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:52,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:52,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:52,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:52,421 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:52,421 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:52,422 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,422 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:52,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:52,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:52,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:52,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aa9d1fa[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:52,428 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:52,428 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,430 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:52,430 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:52,431 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,431 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:52,432 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:52,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:52,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:52,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:52,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:52,435 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:52,435 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:52,435 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:52,435 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:52,435 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:52,437 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,437 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:52,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,437 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:52,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:52,443 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,444 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/9be951fda7c44d1db8edae5c47fb1090 2023-04-01 14:53:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,451 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:52,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/cc8bfc12d6064eb58a865cf1061ae2cd as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/cc8bfc12d6064eb58a865cf1061ae2cd 2023-04-01 14:53:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:52,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:52,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:52,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:52,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:52,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:52,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:52,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:52,464 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:52,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:52,464 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:52,464 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:52,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/cc8bfc12d6064eb58a865cf1061ae2cd, entries=200, sequenceid=346, filesize=14.3 K 2023-04-01 14:53:52,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/c4a5c024c2884fa7a47fcb86c0e99d06 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4a5c024c2884fa7a47fcb86c0e99d06 2023-04-01 14:53:52,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4a5c024c2884fa7a47fcb86c0e99d06, entries=150, sequenceid=346, filesize=11.9 K 2023-04-01 14:53:52,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/9be951fda7c44d1db8edae5c47fb1090 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9be951fda7c44d1db8edae5c47fb1090 2023-04-01 14:53:52,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9be951fda7c44d1db8edae5c47fb1090, entries=150, sequenceid=346, filesize=11.9 K 2023-04-01 14:53:52,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~365.41 KB/374176, currentSize=53.67 KB/54960 for 8f567672fffd4bc77debf33ecd862db3 in 976ms, sequenceid=346, compaction requested=false 2023-04-01 14:53:52,493 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:52,564 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:52,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:52,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:52,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:52,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:52,572 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:52,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:52,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:52,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:52,573 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,573 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,575 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:52,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:52,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:52,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:52,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:52,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:52,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:52,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:52,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:52,592 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:52,592 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:52,593 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-01 14:53:52,593 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:52,593 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:52,593 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:52,593 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:52,593 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:52,593 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:52,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:52,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:52,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360832678, max timestamp=1680360832715Num uniques 859; 2023-04-01 14:53:52,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360832678, max timestamp=1680360832715Num uniques 859; 2023-04-01 14:53:52,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:53:52,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:53:52,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360832678, max timestamp=1680360832715Num uniques 859; 2023-04-01 14:53:52,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:53:52,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360892738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360892744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360892744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360892752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360892848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360892848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360892853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:52,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:52,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360892854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,010 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/bebac4fd62e34219be52f933b4813b82 2023-04-01 14:53:53,026 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/6bb2939464d149aeab53b692ec54c9b0 2023-04-01 14:53:53,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360893053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360893054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360893058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360893059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360893358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360893360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360893363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360893364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,440 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/a8402536d4304bad8220e101ef3b5200 2023-04-01 14:53:53,448 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/bebac4fd62e34219be52f933b4813b82 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bebac4fd62e34219be52f933b4813b82 2023-04-01 14:53:53,457 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bebac4fd62e34219be52f933b4813b82, entries=150, sequenceid=360, filesize=11.9 K 2023-04-01 14:53:53,459 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/6bb2939464d149aeab53b692ec54c9b0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6bb2939464d149aeab53b692ec54c9b0 2023-04-01 14:53:53,465 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6bb2939464d149aeab53b692ec54c9b0, entries=150, sequenceid=360, filesize=11.9 K 2023-04-01 14:53:53,466 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/a8402536d4304bad8220e101ef3b5200 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a8402536d4304bad8220e101ef3b5200 2023-04-01 14:53:53,474 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a8402536d4304bad8220e101ef3b5200, entries=150, sequenceid=360, filesize=11.9 K 2023-04-01 14:53:53,475 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for 8f567672fffd4bc77debf33ecd862db3 in 883ms, sequenceid=360, compaction requested=true 2023-04-01 14:53:53,475 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:53,475 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:53,477 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:53,477 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:53,478 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:53,478 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:53,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:53,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c528667[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:53,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:53,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:53,481 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:53,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:53,481 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:53,481 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:53,481 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:53,482 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,482 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:53,483 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,483 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:53,483 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:53,483 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:53,487 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:53,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:53,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:53,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:53,487 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:53,490 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:53,490 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:53,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:53,490 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:53,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:53,491 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,493 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,496 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:53,496 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:53,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:53,496 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:53,496 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:53,496 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:53,496 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,497 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:53,497 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:53,497 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:53,499 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:53,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:53,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:53,499 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:53,499 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:53,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:53,599 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:53,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:53,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:53,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:53,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:53,604 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:53,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:53,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:53,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:53,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:53,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:53,605 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:53,605 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:53,606 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,606 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:53,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:53,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:53,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:53,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:53,607 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:53,607 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:53,607 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:53,607 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:53,607 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:53,607 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:53,607 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:53,608 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-01 14:53:53,608 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:53,608 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:53,608 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:53,608 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:53,608 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:53,608 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:53,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:53,871 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:53,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360893883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360893886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360893887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360893888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360893990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360893991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360893992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:53,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:53,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360893992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,023 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/d711be7a983b4064a495cc9cab471606 2023-04-01 14:53:54,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:54,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360894194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:54,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360894194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:54,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:54,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360894195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360894196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,443 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/5c2d89246c37449da664a9493df1d2a7 2023-04-01 14:53:54,459 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/ac460c0327ed43d3ac92260eb419d69f 2023-04-01 14:53:54,467 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/d711be7a983b4064a495cc9cab471606 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d711be7a983b4064a495cc9cab471606 2023-04-01 14:53:54,474 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d711be7a983b4064a495cc9cab471606, entries=150, sequenceid=396, filesize=11.9 K 2023-04-01 14:53:54,475 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/5c2d89246c37449da664a9493df1d2a7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5c2d89246c37449da664a9493df1d2a7 2023-04-01 14:53:54,493 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5c2d89246c37449da664a9493df1d2a7, entries=150, sequenceid=396, filesize=11.9 K 2023-04-01 14:53:54,494 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/ac460c0327ed43d3ac92260eb419d69f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/ac460c0327ed43d3ac92260eb419d69f 2023-04-01 14:53:54,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:54,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360894498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:54,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51672 deadline: 1680360894498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:54,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360894498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:54,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360894499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,505 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/ac460c0327ed43d3ac92260eb419d69f, entries=150, sequenceid=396, filesize=11.9 K 2023-04-01 14:53:54,506 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=46.96 KB/48090 for 8f567672fffd4bc77debf33ecd862db3 in 898ms, sequenceid=396, compaction requested=true 2023-04-01 14:53:54,506 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:54,506 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:54,506 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:54,507 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:54,507 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,507 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:54,507 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:54,509 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:54,509 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:54,509 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:54,512 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:54,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:54,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7db2e8ab[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:54,514 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:54,514 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:54,515 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,515 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:54,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:54,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:54,518 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,518 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:54,518 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:54,518 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:54,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:54,521 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:54,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:54,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:54,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:54,521 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:54,522 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:54,522 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:54,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:54,522 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:54,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:54,523 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,524 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:54,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:54,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:54,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:54,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:54,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:54,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:54,531 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:54,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:54,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:54,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:54,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:54,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:54,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:54,632 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:54,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:54,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:54,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:54,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:54,636 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:54,636 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:54,636 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:54,636 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:54,637 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:54,637 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:54,638 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:54,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:54,638 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:54,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:54,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:54,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:54,640 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:54,640 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:54,640 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:54,640 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:54,640 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:54,640 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:54,640 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:54,640 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:54,640 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:54,640 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:53:54,641 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:54,641 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:54,641 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:54,641 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:54,641 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:54,641 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:55,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:55,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:55,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360835017, max timestamp=1680360835029Num uniques 859; 2023-04-01 14:53:55,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:53:55,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360835017, max timestamp=1680360835029Num uniques 859; 2023-04-01 14:53:55,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:53:55,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360835017, max timestamp=1680360835029Num uniques 859; 2023-04-01 14:53:55,031 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:53:55,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360895052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360895053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360895053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360895054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,068 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/b534248419ca4774844b16b3a4af06fb 2023-04-01 14:53:55,093 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/cef7763c72344ec4b9d7886ac908ebff 2023-04-01 14:53:55,115 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/20266795213245918556a040ca78a4b0 2023-04-01 14:53:55,122 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/b534248419ca4774844b16b3a4af06fb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b534248419ca4774844b16b3a4af06fb 2023-04-01 14:53:55,129 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b534248419ca4774844b16b3a4af06fb, entries=150, sequenceid=406, filesize=11.9 K 2023-04-01 14:53:55,131 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/cef7763c72344ec4b9d7886ac908ebff as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/cef7763c72344ec4b9d7886ac908ebff 2023-04-01 14:53:55,136 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/cef7763c72344ec4b9d7886ac908ebff, entries=150, sequenceid=406, filesize=11.9 K 2023-04-01 14:53:55,137 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/20266795213245918556a040ca78a4b0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20266795213245918556a040ca78a4b0 2023-04-01 14:53:55,143 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20266795213245918556a040ca78a4b0, entries=150, sequenceid=406, filesize=11.9 K 2023-04-01 14:53:55,144 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 8f567672fffd4bc77debf33ecd862db3 in 504ms, sequenceid=406, compaction requested=true 2023-04-01 14:53:55,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:55,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:55,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:55,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:55,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@456da571[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:55,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:55,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,151 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:55,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:55,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:55,151 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,151 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:55,152 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,154 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:55,154 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,154 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:55,157 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:55,157 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:55,157 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:55,157 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:55,157 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:55,160 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:55,160 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,160 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-01 14:53:55,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,160 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:55,160 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,161 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:55,161 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:55,161 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:55,161 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:55,161 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,164 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,166 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:55,166 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,166 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,166 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,166 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:55,166 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:55,166 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:55,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:55,166 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,166 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,166 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,166 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:55,166 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:55,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360895169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360895169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360895170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360895171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/b6dc69b449fb4b1baaa443fe3d1acbb7 2023-04-01 14:53:55,194 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/34035bc684cd47da915ddf9d267a9ecb 2023-04-01 14:53:55,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51686 deadline: 1680360895197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,199 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8198 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:53:55,212 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/d11690b05bfd476e9ecf01c4aaa1bb70 2023-04-01 14:53:55,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/b6dc69b449fb4b1baaa443fe3d1acbb7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b6dc69b449fb4b1baaa443fe3d1acbb7 2023-04-01 14:53:55,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b6dc69b449fb4b1baaa443fe3d1acbb7, entries=150, sequenceid=443, filesize=11.9 K 2023-04-01 14:53:55,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/34035bc684cd47da915ddf9d267a9ecb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/34035bc684cd47da915ddf9d267a9ecb 2023-04-01 14:53:55,229 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/34035bc684cd47da915ddf9d267a9ecb, entries=150, sequenceid=443, filesize=11.9 K 2023-04-01 14:53:55,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/d11690b05bfd476e9ecf01c4aaa1bb70 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d11690b05bfd476e9ecf01c4aaa1bb70 2023-04-01 14:53:55,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d11690b05bfd476e9ecf01c4aaa1bb70, entries=150, sequenceid=443, filesize=11.9 K 2023-04-01 14:53:55,239 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 8f567672fffd4bc77debf33ecd862db3 in 79ms, sequenceid=443, compaction requested=true 2023-04-01 14:53:55,239 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:55,239 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:55,239 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:55,239 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 14:53:55,239 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 14:53:55,239 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:55,241 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76170 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 14:53:55,241 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/A is initiating minor compaction (all files) 2023-04-01 14:53:55,241 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73730 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 14:53:55,242 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/A in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,242 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/B is initiating minor compaction (all files) 2023-04-01 14:53:55,242 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/01b9266a0a644957bf05a6d0b4092dc7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/cc8bfc12d6064eb58a865cf1061ae2cd, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bebac4fd62e34219be52f933b4813b82, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d711be7a983b4064a495cc9cab471606, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b534248419ca4774844b16b3a4af06fb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b6dc69b449fb4b1baaa443fe3d1acbb7] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=74.4 K 2023-04-01 14:53:55,242 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/B in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,242 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/429ccab98ddc48abbff9ae5c828e5af8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4a5c024c2884fa7a47fcb86c0e99d06, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6bb2939464d149aeab53b692ec54c9b0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5c2d89246c37449da664a9493df1d2a7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/cef7763c72344ec4b9d7886ac908ebff, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/34035bc684cd47da915ddf9d267a9ecb] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=72.0 K 2023-04-01 14:53:55,242 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 01b9266a0a644957bf05a6d0b4092dc7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680360830717 2023-04-01 14:53:55,242 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 429ccab98ddc48abbff9ae5c828e5af8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680360830717 2023-04-01 14:53:55,243 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting cc8bfc12d6064eb58a865cf1061ae2cd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680360831384 2023-04-01 14:53:55,243 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c4a5c024c2884fa7a47fcb86c0e99d06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680360831384 2023-04-01 14:53:55,243 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting bebac4fd62e34219be52f933b4813b82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680360831520 2023-04-01 14:53:55,243 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6bb2939464d149aeab53b692ec54c9b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680360831520 2023-04-01 14:53:55,245 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d711be7a983b4064a495cc9cab471606, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680360832727 2023-04-01 14:53:55,245 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting b534248419ca4774844b16b3a4af06fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680360833869 2023-04-01 14:53:55,246 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting b6dc69b449fb4b1baaa443fe3d1acbb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680360835052 2023-04-01 14:53:55,247 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 5c2d89246c37449da664a9493df1d2a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680360832727 2023-04-01 14:53:55,251 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cef7763c72344ec4b9d7886ac908ebff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680360833869 2023-04-01 14:53:55,252 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 34035bc684cd47da915ddf9d267a9ecb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680360835052 2023-04-01 14:53:55,267 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:55,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:55,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:55,272 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#A#compaction#72 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:55,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:55,278 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:53:55,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:55,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:55,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:55,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:55,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:55,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:55,279 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#B#compaction#73 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:55,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:55,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:55,292 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:55,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:55,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:55,293 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:55,293 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,293 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,295 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,295 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:55,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:55,295 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,295 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:55,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:55,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:55,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:55,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:55,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:55,315 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,315 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:55,315 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:55,316 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:55,316 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,318 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/66340225efbf4da68f8135fff0669f91 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/66340225efbf4da68f8135fff0669f91 2023-04-01 14:53:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:55,326 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,327 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,327 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:55,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@515f8a27[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:55,330 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:55,330 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,331 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,331 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:55,331 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:55,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:55,331 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:55,333 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,335 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/A of 8f567672fffd4bc77debf33ecd862db3 into 66340225efbf4da68f8135fff0669f91(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:55,335 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,335 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/A, priority=10, startTime=1680360835239; duration=0sec 2023-04-01 14:53:55,335 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:55,335 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 14:53:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,338 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73730 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 14:53:55,338 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/C is initiating minor compaction (all files) 2023-04-01 14:53:55,338 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/C in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,338 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/c61e51d14246431a82a481f86edc4f8e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9be951fda7c44d1db8edae5c47fb1090, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a8402536d4304bad8220e101ef3b5200, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/ac460c0327ed43d3ac92260eb419d69f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20266795213245918556a040ca78a4b0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d11690b05bfd476e9ecf01c4aaa1bb70] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=72.0 K 2023-04-01 14:53:55,339 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting c61e51d14246431a82a481f86edc4f8e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680360830717 2023-04-01 14:53:55,339 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 9be951fda7c44d1db8edae5c47fb1090, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680360831384 2023-04-01 14:53:55,339 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting a8402536d4304bad8220e101ef3b5200, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680360831520 2023-04-01 14:53:55,340 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting ac460c0327ed43d3ac92260eb419d69f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680360832727 2023-04-01 14:53:55,340 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 20266795213245918556a040ca78a4b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680360833869 2023-04-01 14:53:55,341 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d11690b05bfd476e9ecf01c4aaa1bb70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680360835052 2023-04-01 14:53:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:55,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:55,346 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:55,346 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:55,346 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:55,346 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:55,346 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:55,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,350 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,351 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:55,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,355 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,355 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#C#compaction#75 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:55,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:55,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:55,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:55,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:55,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,356 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:55,356 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,356 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:55,356 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,356 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360835344, max timestamp=1680360835354Num uniques 859; 2023-04-01 14:53:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360835344, max timestamp=1680360835353Num uniques 859; 2023-04-01 14:53:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360835344, max timestamp=1680360835353Num uniques 859; 2023-04-01 14:53:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:53:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:53:55,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:53:55,377 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/b8ea6a682fd846c6b3de10239b856071 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b8ea6a682fd846c6b3de10239b856071 2023-04-01 14:53:55,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360895377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360895380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,384 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/C of 8f567672fffd4bc77debf33ecd862db3 into b8ea6a682fd846c6b3de10239b856071(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:55,384 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:55,384 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/C, priority=10, startTime=1680360835239; duration=0sec 2023-04-01 14:53:55,384 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:55,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360895381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360895382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,456 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:55,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:55,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:55,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:55,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:55,460 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:55,460 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:55,460 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:55,460 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:55,461 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,461 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:55,462 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:55,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:55,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:55,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:55,464 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:55,464 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:55,464 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:55,464 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:55,464 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:55,464 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,464 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:55,465 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:55,465 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:55,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,466 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:55,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59365fd5[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:55,469 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:55,469 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,470 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:55,470 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,470 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:55,475 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:55,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:55,478 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:55,478 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:55,478 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:55,478 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:55,478 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:55,479 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,479 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,479 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,483 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360895483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360895484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,486 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:55,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:55,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:55,487 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:55,487 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:55,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:55,487 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,487 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,487 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:55,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360895488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360895488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,587 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:55,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:55,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:55,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:55,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:55,591 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:55,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:55,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:55,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:55,592 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,592 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,597 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:55,597 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:55,597 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,597 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:55,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:55,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:55,600 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,600 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:55,600 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:55,600 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:55,601 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,601 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,602 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,602 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:55,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@332b13e8[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:55,603 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:55,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,605 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,605 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,605 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:55,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:55,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:55,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:55,610 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:55,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:55,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:55,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:55,611 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:55,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,613 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,614 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,617 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,617 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,617 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,617 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:55,617 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,617 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,617 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,617 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,618 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,618 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:55,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:55,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:55,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:55,618 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:55,618 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:55,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360895686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360895686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360895692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360895693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,716 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/1f327c36c28148179c6834ee02e18ad8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/1f327c36c28148179c6834ee02e18ad8 2023-04-01 14:53:55,718 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:55,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:55,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:55,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:55,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:55,724 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:55,724 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:55,724 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:55,724 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:55,728 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,728 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,728 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/B of 8f567672fffd4bc77debf33ecd862db3 into 1f327c36c28148179c6834ee02e18ad8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:55,729 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:55,729 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/B, priority=10, startTime=1680360835239; duration=0sec 2023-04-01 14:53:55,729 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/b74b834f3110443f831d09787e470ab7 2023-04-01 14:53:55,729 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:55,730 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:55,730 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:55,731 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,731 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:55,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:55,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:55,744 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,744 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:55,745 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:55,745 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:55,747 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,747 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:55,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b456ed1[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:55,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:55,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,752 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:55,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:55,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:55,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:55,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:55,756 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:55,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:55,761 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:55,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:55,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:55,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:55,761 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:55,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,764 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,764 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,766 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,775 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:55,775 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:55,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:55,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:55,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:55,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,776 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:55,777 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:55,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/2e217420341d44bab2ad99c62b5b5d38 2023-04-01 14:53:55,877 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:55,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:55,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:55,880 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:55,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:55,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:55,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:55,881 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,881 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,882 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:55,882 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:55,882 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,882 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:55,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:55,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:55,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:55,884 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,884 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:55,885 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:55,885 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:55,885 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:55,885 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:55,885 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,885 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:55,885 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:55,886 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,886 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,886 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,886 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:55,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36e6c12c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:55,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:55,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,890 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,891 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:55,891 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:55,891 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:55,891 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,891 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:55,893 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:55,893 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,893 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:55,894 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:55,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:55,899 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:55,899 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:55,899 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:55,899 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:55,899 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:55,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:55,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:55,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,904 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:55,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:55,909 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,909 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,909 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:55,909 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,909 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:55,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:55,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:55,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:55,909 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,909 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:55,909 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:55,909 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:55,909 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:55,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:55,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360895990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360895990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360895996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:55,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:55,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360895998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,009 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:56,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:56,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:56,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:56,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:56,015 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:56,015 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:56,015 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:56,015 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:56,016 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,016 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,017 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,017 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:56,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:56,018 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,018 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:56,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:56,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:56,021 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:56,021 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:56,021 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:56,021 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:56,021 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:56,021 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:56,021 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:56,021 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:56,021 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,022 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:56,022 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:56,023 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,023 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:56,023 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,023 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:56,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:56,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dc5f8d2[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:56,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:56,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,031 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,031 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:56,031 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:56,031 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:56,031 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,031 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:56,033 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:56,033 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:56,036 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:56,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:56,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:56,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:56,036 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:56,037 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,037 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,037 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:56,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:56,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:56,038 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:56,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:56,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:56,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:56,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,042 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:56,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:56,042 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:56,043 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:56,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:56,143 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:56,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:56,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:56,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:56,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:56,148 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:56,148 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:56,148 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:56,148 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:56,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,150 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:56,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:56,150 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,151 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,151 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:56,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:56,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:56,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:56,155 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:56,155 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:56,155 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:56,160 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:56,160 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:56,160 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:56,160 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:56,160 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:56,160 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:56,161 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:56,161 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:56,161 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,161 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:56,161 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:56,169 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,169 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:56,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:56,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:56,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@369a04d1[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:56,174 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:56,174 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,177 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:56,179 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,179 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:56,181 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:56,186 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:56,186 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:56,186 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:56,186 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:56,186 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:56,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:56,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,196 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,198 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:56,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:56,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:56,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:56,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:56,203 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:56,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:56,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:56,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,203 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:56,203 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:56,219 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/f6e9185711ac45578d039c8fc3b3c769 2023-04-01 14:53:56,227 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/b74b834f3110443f831d09787e470ab7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b74b834f3110443f831d09787e470ab7 2023-04-01 14:53:56,234 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b74b834f3110443f831d09787e470ab7, entries=300, sequenceid=454, filesize=19.0 K 2023-04-01 14:53:56,236 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/2e217420341d44bab2ad99c62b5b5d38 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/2e217420341d44bab2ad99c62b5b5d38 2023-04-01 14:53:56,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/2e217420341d44bab2ad99c62b5b5d38, entries=150, sequenceid=454, filesize=11.9 K 2023-04-01 14:53:56,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/f6e9185711ac45578d039c8fc3b3c769 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/f6e9185711ac45578d039c8fc3b3c769 2023-04-01 14:53:56,253 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/f6e9185711ac45578d039c8fc3b3c769, entries=150, sequenceid=454, filesize=11.9 K 2023-04-01 14:53:56,254 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8f567672fffd4bc77debf33ecd862db3 in 976ms, sequenceid=454, compaction requested=false 2023-04-01 14:53:56,254 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:56,304 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:56,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:56,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:56,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:56,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:56,309 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:56,309 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:56,309 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:56,309 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:56,310 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,310 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,312 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,312 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:56,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,313 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,313 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:56,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:56,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:56,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:56,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:56,330 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:56,330 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:56,331 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-01 14:53:56,331 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:56,331 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:56,331 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:56,331 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:56,331 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:56,332 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:56,351 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/2ccd7d2081d243c1b706e8b6b1f1ff29 2023-04-01 14:53:56,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:56,496 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:56,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360896505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360896506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360896508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360896508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360896609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360896609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360896610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360896610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,774 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/400fa5b54b8544bc9f016ad5b883ccaa 2023-04-01 14:53:56,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360896815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360896815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360896816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:56,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:56,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360896816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360897119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360897120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360897122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360897122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,191 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/1aa88a34713945858701e04e42560881 2023-04-01 14:53:57,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/2ccd7d2081d243c1b706e8b6b1f1ff29 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/2ccd7d2081d243c1b706e8b6b1f1ff29 2023-04-01 14:53:57,206 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/2ccd7d2081d243c1b706e8b6b1f1ff29, entries=150, sequenceid=493, filesize=11.9 K 2023-04-01 14:53:57,208 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/400fa5b54b8544bc9f016ad5b883ccaa as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/400fa5b54b8544bc9f016ad5b883ccaa 2023-04-01 14:53:57,216 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/400fa5b54b8544bc9f016ad5b883ccaa, entries=150, sequenceid=493, filesize=11.9 K 2023-04-01 14:53:57,223 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/1aa88a34713945858701e04e42560881 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/1aa88a34713945858701e04e42560881 2023-04-01 14:53:57,244 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/1aa88a34713945858701e04e42560881, entries=150, sequenceid=493, filesize=11.9 K 2023-04-01 14:53:57,245 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 8f567672fffd4bc77debf33ecd862db3 in 914ms, sequenceid=493, compaction requested=true 2023-04-01 14:53:57,246 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:57,246 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:57,246 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:57,246 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:57,246 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:57,248 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:57,249 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:57,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:57,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:57,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:57,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@619d735[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:57,253 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,254 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:57,254 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:57,254 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:57,254 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,254 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:57,256 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,256 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:57,256 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:57,256 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:57,259 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:57,259 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:57,259 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:57,259 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:57,259 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:57,261 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,261 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:57,261 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,261 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:57,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:57,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:57,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,263 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:57,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:57,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:57,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:57,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:57,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:57,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:57,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,266 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:57,266 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:57,266 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:57,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:57,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:57,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:57,366 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:57,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:57,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:57,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:57,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:57,370 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:57,370 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:57,370 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:57,370 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:57,371 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,371 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:57,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:57,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:57,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:57,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,374 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:57,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:57,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:57,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:57,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:57,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:57,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:57,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:57,376 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:57,376 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:57,376 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:57,376 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:57,376 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 14:53:57,376 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:57,376 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:57,376 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:57,376 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:57,377 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:57,377 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:57,390 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/c8ec1d442a9e4a9b92c70ba6dc3caae7 2023-04-01 14:53:57,412 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/dec371d528074c74995edf109f43d3f1 2023-04-01 14:53:57,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:57,625 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:57,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360897643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680360837636, max timestamp=1680360837646Num uniques 859; 2023-04-01 14:53:57,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:53:57,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680360837636, max timestamp=1680360837646Num uniques 859; 2023-04-01 14:53:57,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680360837636, max timestamp=1680360837646Num uniques 859; 2023-04-01 14:53:57,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:53:57,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 14:53:57,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360897686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360897691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360897691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360897750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360897792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360897795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360897798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,827 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/d6cfe478fe454af3893e5793d939f7af 2023-04-01 14:53:57,835 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/c8ec1d442a9e4a9b92c70ba6dc3caae7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c8ec1d442a9e4a9b92c70ba6dc3caae7 2023-04-01 14:53:57,841 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c8ec1d442a9e4a9b92c70ba6dc3caae7, entries=150, sequenceid=505, filesize=11.9 K 2023-04-01 14:53:57,842 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/dec371d528074c74995edf109f43d3f1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dec371d528074c74995edf109f43d3f1 2023-04-01 14:53:57,848 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dec371d528074c74995edf109f43d3f1, entries=150, sequenceid=505, filesize=11.9 K 2023-04-01 14:53:57,850 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/d6cfe478fe454af3893e5793d939f7af as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d6cfe478fe454af3893e5793d939f7af 2023-04-01 14:53:57,856 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d6cfe478fe454af3893e5793d939f7af, entries=150, sequenceid=505, filesize=11.9 K 2023-04-01 14:53:57,861 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=124.84 KB/127836 for 8f567672fffd4bc77debf33ecd862db3 in 485ms, sequenceid=505, compaction requested=true 2023-04-01 14:53:57,861 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:57,861 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:57,862 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:57,864 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6897b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:57,866 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:57,866 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,868 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,868 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:57,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:57,873 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:57,873 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:57,873 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,873 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:57,877 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:57,877 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:57,877 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:57,877 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:57,877 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:57,880 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:57,881 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,882 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,884 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,884 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:57,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,888 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,888 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:57,888 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:57,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:57,888 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:57,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:57,888 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,888 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:57,889 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:57,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:57,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:57,889 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:57,889 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:57,889 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:57,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:57,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:57,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=131.55 KB heapSize=374.30 KB 2023-04-01 14:53:57,954 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:57,954 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:57,954 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:57,955 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:57,955 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:57,955 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:57,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:57,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360897975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,989 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:57,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:57,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:57,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:57,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:57,993 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:57,993 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:57,994 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:57,994 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:57,994 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,994 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:57,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:57,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:57,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:57,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:57,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:57,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:57,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:57,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:57,998 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:57,998 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:57,998 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:57,999 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:57,999 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:57,999 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:57,999 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:58,000 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:58,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360897999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360897999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360898001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,001 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,001 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:58,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7407f21f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:58,005 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:58,005 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,007 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:58,007 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,007 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:58,008 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:58,008 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,008 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,008 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:58,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:58,011 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:58,011 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:58,011 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:58,011 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:58,011 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:58,013 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,015 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,016 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,016 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,017 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,019 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,019 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,019 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:58,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:58,019 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,020 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:58,020 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:58,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360898077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,120 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:58,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:58,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:58,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:58,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:58,125 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:58,125 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,125 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:58,125 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:58,125 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,125 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,128 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:58,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:58,129 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:58,129 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,130 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:58,130 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:58,130 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:58,130 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,130 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:58,130 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:58,130 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,130 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:58,130 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:58,130 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,130 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:58,130 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:58,132 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125dccc2[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:58,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:58,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,136 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:58,136 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:58,136 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:58,136 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,136 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:58,137 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,137 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:58,137 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,137 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:58,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:58,140 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:58,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:58,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:58,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:58,140 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:58,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,144 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,146 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,148 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,148 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,148 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,148 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,148 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:58,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:58,148 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,148 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,148 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,148 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,148 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,148 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:58,149 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:58,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,249 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:58,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:58,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:58,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:58,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:58,252 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:58,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:58,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:58,253 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,253 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,254 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,254 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,254 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,254 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:58,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:58,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:58,256 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:58,256 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,256 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:58,256 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:58,256 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:58,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:58,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:58,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,256 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:58,257 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:58,257 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,257 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:58,257 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:58,262 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,262 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,262 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,262 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:58,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab501c3[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:58,264 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:58,264 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:58,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:58,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:58,267 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,267 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:58,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:58,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:58,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:58,271 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:58,271 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:58,271 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:58,271 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:58,271 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:58,273 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,273 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,273 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,274 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,283 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:58,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:58,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,283 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:58,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,283 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:58,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360898282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360898302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360898304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360898304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,365 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/61ddf7447bc44042a84bb255ac5f2e1a 2023-04-01 14:53:58,384 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:58,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:58,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:58,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:58,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:58,387 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:58,387 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,387 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:58,387 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:58,388 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,388 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,389 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,390 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,390 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:58,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:58,395 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:58,395 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,395 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:58,397 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,397 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:58,397 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:58,397 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:58,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,399 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:58,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@683af6bf[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:58,401 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:58,401 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,402 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,402 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:58,402 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:58,402 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:58,402 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,402 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:58,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:58,405 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:58,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:58,409 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:58,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:58,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:58,410 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:58,410 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:58,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,412 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,413 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:58,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:58,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,417 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:58,417 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:58,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:58,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,419 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,419 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,518 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:58,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:58,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:58,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:58,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:58,521 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:58,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:58,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:58,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,529 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,529 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,529 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,529 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:58,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:58,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:58,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:58,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:58,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:58,530 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,530 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:58,531 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:58,531 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:58,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:58,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:58,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:58,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,532 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:58,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23e676ad[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:58,534 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:58,534 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,535 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:58,535 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,535 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:58,536 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,536 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,536 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,537 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:58,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:58,539 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:58,539 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:58,539 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:58,539 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:58,539 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:58,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,542 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,544 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,546 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:58,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:58,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:58,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,547 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:58,547 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:58,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360898586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,647 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:58,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:58,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:58,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:58,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:58,651 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:58,651 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,651 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:58,651 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:58,652 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,652 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,653 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,653 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,653 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,653 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:58,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:58,654 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:58,654 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,654 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:58,656 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,656 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:58,656 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:58,656 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:58,659 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,659 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,659 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,659 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:58,662 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:58,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fa185cf[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:58,662 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,663 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,663 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:58,663 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:58,663 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:58,663 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,663 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:58,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,665 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:58,665 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,665 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:58,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:58,669 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:58,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:58,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:58,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:58,669 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:58,672 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,672 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,672 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,672 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,673 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:58,678 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:58,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,679 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:58,679 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:58,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,779 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/ade23a6539324592aedd63279a3d5063 2023-04-01 14:53:58,779 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:58,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:58,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:58,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:58,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:58,786 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:58,786 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,786 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:58,786 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:58,787 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,787 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,788 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,788 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,788 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,788 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:58,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:58,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:58,789 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:58,789 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:58,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:58,790 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,790 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:58,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:58,790 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:58,790 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:58,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:58,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:58,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:58,793 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,793 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,794 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,794 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:58,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d683ce1[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:58,799 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:58,799 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,800 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,800 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:58,800 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:58,800 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:58,800 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,800 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:58,801 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.10 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/60ed58535b604a9e9bdc25bf0949a679 2023-04-01 14:53:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,802 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360898807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:58,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:58,808 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:58,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:58,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:58,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:58,808 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:58,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/61ddf7447bc44042a84bb255ac5f2e1a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61ddf7447bc44042a84bb255ac5f2e1a 2023-04-01 14:53:58,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360898809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:58,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360898812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,815 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:58,815 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,815 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:58,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,823 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:58,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61ddf7447bc44042a84bb255ac5f2e1a, entries=200, sequenceid=541, filesize=14.3 K 2023-04-01 14:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/ade23a6539324592aedd63279a3d5063 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/ade23a6539324592aedd63279a3d5063 2023-04-01 14:53:58,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,827 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:58,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:58,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:58,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:58,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:58,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,828 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:58,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:58,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:58,828 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:58,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/ade23a6539324592aedd63279a3d5063, entries=150, sequenceid=541, filesize=11.9 K 2023-04-01 14:53:58,845 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/60ed58535b604a9e9bdc25bf0949a679 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/60ed58535b604a9e9bdc25bf0949a679 2023-04-01 14:53:58,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/60ed58535b604a9e9bdc25bf0949a679, entries=150, sequenceid=541, filesize=11.9 K 2023-04-01 14:53:58,852 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.55 KB/134706, heapSize ~374.26 KB/383240, currentSize=46.96 KB/48090 for 8f567672fffd4bc77debf33ecd862db3 in 898ms, sequenceid=541, compaction requested=true 2023-04-01 14:53:58,852 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:58,852 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:58,852 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:53:58,852 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:53:58,852 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:58,852 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:58,854 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 71508 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:53:58,854 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/A is initiating minor compaction (all files) 2023-04-01 14:53:58,854 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/A in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,854 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/66340225efbf4da68f8135fff0669f91, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b74b834f3110443f831d09787e470ab7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/2ccd7d2081d243c1b706e8b6b1f1ff29, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c8ec1d442a9e4a9b92c70ba6dc3caae7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61ddf7447bc44042a84bb255ac5f2e1a] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=69.8 K 2023-04-01 14:53:58,855 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:53:58,855 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 66340225efbf4da68f8135fff0669f91, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680360835052 2023-04-01 14:53:58,855 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/B is initiating minor compaction (all files) 2023-04-01 14:53:58,855 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/B in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,855 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/1f327c36c28148179c6834ee02e18ad8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/2e217420341d44bab2ad99c62b5b5d38, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/400fa5b54b8544bc9f016ad5b883ccaa, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dec371d528074c74995edf109f43d3f1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/ade23a6539324592aedd63279a3d5063] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=60.3 K 2023-04-01 14:53:58,855 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting b74b834f3110443f831d09787e470ab7, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680360835161 2023-04-01 14:53:58,856 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1f327c36c28148179c6834ee02e18ad8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680360835052 2023-04-01 14:53:58,856 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 2ccd7d2081d243c1b706e8b6b1f1ff29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680360835375 2023-04-01 14:53:58,856 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting c8ec1d442a9e4a9b92c70ba6dc3caae7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680360836496 2023-04-01 14:53:58,856 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2e217420341d44bab2ad99c62b5b5d38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680360835275 2023-04-01 14:53:58,857 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 61ddf7447bc44042a84bb255ac5f2e1a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680360837685 2023-04-01 14:53:58,857 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 400fa5b54b8544bc9f016ad5b883ccaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680360835375 2023-04-01 14:53:58,857 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting dec371d528074c74995edf109f43d3f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680360836496 2023-04-01 14:53:58,858 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ade23a6539324592aedd63279a3d5063, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680360837685 2023-04-01 14:53:58,871 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#B#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:58,874 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#A#compaction#88 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:58,928 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:58,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:58,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:58,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:58,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:58,932 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:58,932 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,932 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:58,932 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:58,933 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,933 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,934 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,934 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:58,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:58,934 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:58,934 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:58,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:58,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:58,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:58,935 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:58,935 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:58,935 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:58,936 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:58,936 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:58,936 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:58,936 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:58,936 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:53:58,936 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:58,936 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:58,936 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:58,937 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:58,937 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:58,937 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:53:59,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:59,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:59,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360839121, max timestamp=1680360839139Num uniques 859; 2023-04-01 14:53:59,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:53:59,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360839121, max timestamp=1680360839139Num uniques 859; 2023-04-01 14:53:59,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:53:59,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360839121, max timestamp=1680360839139Num uniques 859; 2023-04-01 14:53:59,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:53:59,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:59,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360899196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,300 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/e8a19b568f4048e8ae0f6b00a48af166 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e8a19b568f4048e8ae0f6b00a48af166 2023-04-01 14:53:59,302 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/21a99ba58e234bdca518931cc36f3c6c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/21a99ba58e234bdca518931cc36f3c6c 2023-04-01 14:53:59,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:59,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360899298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,314 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/B of 8f567672fffd4bc77debf33ecd862db3 into e8a19b568f4048e8ae0f6b00a48af166(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:59,314 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:59,314 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/B, priority=11, startTime=1680360838852; duration=0sec 2023-04-01 14:53:59,314 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:53:59,314 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:53:59,316 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:53:59,316 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/C is initiating minor compaction (all files) 2023-04-01 14:53:59,317 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/C in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:59,317 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b8ea6a682fd846c6b3de10239b856071, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/f6e9185711ac45578d039c8fc3b3c769, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/1aa88a34713945858701e04e42560881, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d6cfe478fe454af3893e5793d939f7af, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/60ed58535b604a9e9bdc25bf0949a679] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=60.3 K 2023-04-01 14:53:59,317 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b8ea6a682fd846c6b3de10239b856071, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680360835052 2023-04-01 14:53:59,318 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f6e9185711ac45578d039c8fc3b3c769, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680360835275 2023-04-01 14:53:59,319 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1aa88a34713945858701e04e42560881, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1680360835375 2023-04-01 14:53:59,322 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/A of 8f567672fffd4bc77debf33ecd862db3 into 21a99ba58e234bdca518931cc36f3c6c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:59,322 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:59,322 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/A, priority=11, startTime=1680360838852; duration=0sec 2023-04-01 14:53:59,322 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:59,322 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting d6cfe478fe454af3893e5793d939f7af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680360836496 2023-04-01 14:53:59,323 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 60ed58535b604a9e9bdc25bf0949a679, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680360837685 2023-04-01 14:53:59,338 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#C#compaction#90 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:53:59,349 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/d5e68b4c76224df68d8296f145434961 2023-04-01 14:53:59,375 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/11d07ac7d28c499694359cafb774947c 2023-04-01 14:53:59,394 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/fba2cdcc921c4f60ab5256cdc0d2b128 2023-04-01 14:53:59,399 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/d5e68b4c76224df68d8296f145434961 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d5e68b4c76224df68d8296f145434961 2023-04-01 14:53:59,406 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d5e68b4c76224df68d8296f145434961, entries=100, sequenceid=551, filesize=9.5 K 2023-04-01 14:53:59,407 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/11d07ac7d28c499694359cafb774947c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/11d07ac7d28c499694359cafb774947c 2023-04-01 14:53:59,414 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/11d07ac7d28c499694359cafb774947c, entries=100, sequenceid=551, filesize=9.5 K 2023-04-01 14:53:59,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/fba2cdcc921c4f60ab5256cdc0d2b128 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/fba2cdcc921c4f60ab5256cdc0d2b128 2023-04-01 14:53:59,439 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/fba2cdcc921c4f60ab5256cdc0d2b128, entries=100, sequenceid=551, filesize=9.5 K 2023-04-01 14:53:59,440 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8f567672fffd4bc77debf33ecd862db3 in 504ms, sequenceid=551, compaction requested=false 2023-04-01 14:53:59,440 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:59,440 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:59,441 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:59,441 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:59,441 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,441 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:59,441 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:59,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,442 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:59,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:59,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12d8a8fd[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:59,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:59,451 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,451 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:59,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:59,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:59,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:59,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:59,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:59,453 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:59,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:59,458 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:59,458 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:59,458 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:59,458 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:59,458 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:59,460 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,460 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,461 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:59,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,462 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,470 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:59,470 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:59,470 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:59,470 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:59,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:59,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:59,470 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,470 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,470 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,470 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,470 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:59,470 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:59,470 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,471 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:59,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:59,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:53:59,515 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-01 14:53:59,515 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:53:59,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:59,515 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:53:59,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:59,515 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:53:59,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:53:59,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:59,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360899553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,571 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:59,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:59,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:59,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:59,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:59,580 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:59,580 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:59,580 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:59,580 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:59,581 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,581 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,583 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,583 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:59,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:59,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,585 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,585 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:59,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:59,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:59,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:59,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:59,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:59,588 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:59,588 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:59,589 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:59,589 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:59,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:59,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:59,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:59,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:59,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:59,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21bf715[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:59,597 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:59,597 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,598 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:59,599 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,599 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:59,601 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:59,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:59,608 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:59,608 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:59,608 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:59,608 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:59,608 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:59,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:59,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,612 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,614 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,621 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:59,621 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:59,621 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,622 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,622 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,622 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,622 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,622 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,622 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:59,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:59,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:59,621 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:59,622 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:59,622 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:59,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:59,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:59,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360899658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,722 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:59,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:59,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:59,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:59,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:59,747 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:59,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:59,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:59,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:59,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,748 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,751 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:59,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:59,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,753 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,753 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:59,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:59,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:59,760 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:59,760 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:59,760 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:59,760 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:59,760 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:59,760 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:59,760 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,760 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:59,760 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:59,762 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/9398eabd8d68406ea8dbe7b9e70c6fad as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9398eabd8d68406ea8dbe7b9e70c6fad 2023-04-01 14:53:59,765 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:59,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51478564[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:59,777 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/C of 8f567672fffd4bc77debf33ecd862db3 into 9398eabd8d68406ea8dbe7b9e70c6fad(size=12.9 K), total size for store is 22.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:53:59,777 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:53:59,777 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/C, priority=11, startTime=1680360838852; duration=0sec 2023-04-01 14:53:59,777 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:59,777 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,777 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:53:59,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,780 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:59,780 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:59,780 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:59,781 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,781 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:59,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,782 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:59,782 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:59,782 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:59,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:59,787 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:59,787 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:59,787 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:59,787 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:59,787 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:59,790 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,790 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:59,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,790 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,792 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,793 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:59,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:59,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,797 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:59,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:59,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:59,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:59,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:59,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:59,798 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:59,798 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:59,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:59,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:59,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360899815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:59,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360899816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:59,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360899819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:53:59,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360899862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,898 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:53:59,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:53:59,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:53:59,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:53:59,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:53:59,902 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:53:59,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:59,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:53:59,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:53:59,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,903 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:59,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:59,904 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,904 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,904 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:53:59,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:53:59,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:53:59,905 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:53:59,905 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:53:59,905 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:53:59,906 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:59,906 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:53:59,906 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:53:59,906 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:53:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:53:59,908 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,909 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,909 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,909 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:53:59,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d242499[Count = 0] remaining members to acquire global barrier 2023-04-01 14:53:59,911 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:53:59,911 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,912 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,912 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:53:59,912 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:53:59,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:53:59,912 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:53:59,914 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,914 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:53:59,914 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,914 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:53:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:53:59,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:53:59,917 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:53:59,917 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:53:59,917 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:53:59,917 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:53:59,917 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:53:59,919 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,919 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:53:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:53:59,919 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:59,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:59,920 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:53:59,922 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:53:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:53:59,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:53:59,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:53:59,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:53:59,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:53:59,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:53:59,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:53:59,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:53:59,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:53:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:53:59,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:53:59,926 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:53:59,927 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:53:59,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:53:59,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:53:59,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:53:59,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/0ca488973b3741299d3002bd2e649dde 2023-04-01 14:53:59,951 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/e6db14164d324e4fa90dbe547d44ea40 2023-04-01 14:54:00,027 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:00,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:00,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:00,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:00,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:00,031 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:00,031 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:00,031 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:00,031 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:00,033 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,033 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:00,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:00,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:00,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:00,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:00,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:00,037 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:00,037 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:54:00,037 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:54:00,037 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:00,037 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:00,038 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:00,040 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:00,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aecfc80[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:00,043 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:00,043 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,044 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:00,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:00,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:00,045 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,045 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:00,046 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:00,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:00,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:00,050 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:00,050 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:00,050 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:00,050 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:00,050 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:00,052 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,052 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:00,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,052 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:00,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:00,053 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,059 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,059 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,059 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,060 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:00,059 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:00,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:00,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:00,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:00,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:00,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,060 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:00,060 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:00,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:00,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:00,160 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:00,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:00,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:00,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:00,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:00,165 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:00,165 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:00,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:00,165 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:00,166 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:00,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360900164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,166 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,166 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,168 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:00,168 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,168 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,168 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:00,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:00,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:00,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:00,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:00,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:00,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:00,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:00,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:00,173 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:00,173 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:54:00,173 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:54:00,173 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:00,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:00,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:00,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:00,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:00,175 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,175 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:00,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:00,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37920d97[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:00,179 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:00,179 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,184 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:00,185 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,185 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:00,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:00,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:00,192 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:00,192 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:00,192 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:00,192 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:00,192 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:00,197 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,197 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,197 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:00,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:00,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,198 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,199 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:00,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:00,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:00,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:00,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:00,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:00,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,202 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:00,202 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:00,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:00,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:00,302 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:00,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:00,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:00,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:00,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:00,306 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:00,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:00,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:00,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:00,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:00,309 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,310 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,310 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:00,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:00,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:00,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:00,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:00,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:00,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:00,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:00,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:00,311 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:00,311 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:54:00,311 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:54:00,311 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:00,313 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:00,313 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:00,313 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:00,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:00,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a13d2a4[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:00,315 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:00,315 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,316 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,316 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:00,316 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:00,316 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:00,316 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,316 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:00,317 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:00,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:00,320 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:00,320 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:00,320 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:00,320 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:00,320 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:00,323 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,323 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:00,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,323 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:00,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:00,324 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,325 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,327 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:00,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:00,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:00,328 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:00,328 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:00,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:00,328 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:00,328 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:00,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:00,367 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/40469b401d9f4782a55ef7fba7fc8d66 2023-04-01 14:54:00,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/0ca488973b3741299d3002bd2e649dde as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/0ca488973b3741299d3002bd2e649dde 2023-04-01 14:54:00,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/0ca488973b3741299d3002bd2e649dde, entries=150, sequenceid=590, filesize=11.9 K 2023-04-01 14:54:00,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/e6db14164d324e4fa90dbe547d44ea40 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e6db14164d324e4fa90dbe547d44ea40 2023-04-01 14:54:00,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e6db14164d324e4fa90dbe547d44ea40, entries=150, sequenceid=590, filesize=11.9 K 2023-04-01 14:54:00,386 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/40469b401d9f4782a55ef7fba7fc8d66 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/40469b401d9f4782a55ef7fba7fc8d66 2023-04-01 14:54:00,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/40469b401d9f4782a55ef7fba7fc8d66, entries=150, sequenceid=590, filesize=11.9 K 2023-04-01 14:54:00,394 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 8f567672fffd4bc77debf33ecd862db3 in 879ms, sequenceid=590, compaction requested=true 2023-04-01 14:54:00,394 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:00,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:00,394 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:00,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:00,394 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:00,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:00,395 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35102 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:00,395 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35102 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:00,395 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/A is initiating minor compaction (all files) 2023-04-01 14:54:00,396 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/A in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:00,396 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/B is initiating minor compaction (all files) 2023-04-01 14:54:00,396 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/21a99ba58e234bdca518931cc36f3c6c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d5e68b4c76224df68d8296f145434961, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/0ca488973b3741299d3002bd2e649dde] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=34.3 K 2023-04-01 14:54:00,396 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/B in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:00,396 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e8a19b568f4048e8ae0f6b00a48af166, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/11d07ac7d28c499694359cafb774947c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e6db14164d324e4fa90dbe547d44ea40] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=34.3 K 2023-04-01 14:54:00,396 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 21a99ba58e234bdca518931cc36f3c6c, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680360837685 2023-04-01 14:54:00,396 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e8a19b568f4048e8ae0f6b00a48af166, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680360837685 2023-04-01 14:54:00,397 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d5e68b4c76224df68d8296f145434961, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680360837966 2023-04-01 14:54:00,397 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 11d07ac7d28c499694359cafb774947c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680360837966 2023-04-01 14:54:00,397 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 0ca488973b3741299d3002bd2e649dde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680360839182 2023-04-01 14:54:00,397 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e6db14164d324e4fa90dbe547d44ea40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680360839182 2023-04-01 14:54:00,407 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#B#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:00,410 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#A#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:00,428 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:00,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:00,431 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/9de69b3ef5d14c44b050b0b93efa28f8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9de69b3ef5d14c44b050b0b93efa28f8 2023-04-01 14:54:00,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:00,438 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/A of 8f567672fffd4bc77debf33ecd862db3 into 9de69b3ef5d14c44b050b0b93efa28f8(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:00,438 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:00,438 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/A, priority=13, startTime=1680360840394; duration=0sec 2023-04-01 14:54:00,438 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:00,439 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:00,440 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35102 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:00,440 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/C is initiating minor compaction (all files) 2023-04-01 14:54:00,440 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/C in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:00,440 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9398eabd8d68406ea8dbe7b9e70c6fad, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/fba2cdcc921c4f60ab5256cdc0d2b128, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/40469b401d9f4782a55ef7fba7fc8d66] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=34.3 K 2023-04-01 14:54:00,440 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 9398eabd8d68406ea8dbe7b9e70c6fad, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680360837685 2023-04-01 14:54:00,441 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting fba2cdcc921c4f60ab5256cdc0d2b128, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680360837966 2023-04-01 14:54:00,441 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 40469b401d9f4782a55ef7fba7fc8d66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680360839182 2023-04-01 14:54:00,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:00,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:00,444 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:00,444 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:00,444 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:00,444 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:00,445 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,445 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,446 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:00,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:00,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:00,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:00,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:00,449 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:00,449 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:00,449 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:00,449 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:54:00,449 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-01 14:54:00,449 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:54:00,450 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:00,450 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/535a160555d947968d734cfb99bbe38c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/535a160555d947968d734cfb99bbe38c 2023-04-01 14:54:00,450 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:54:00,450 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:00,450 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:54:00,450 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:00,451 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#C#compaction#98 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:00,462 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/B of 8f567672fffd4bc77debf33ecd862db3 into 535a160555d947968d734cfb99bbe38c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:00,462 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:00,462 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/B, priority=13, startTime=1680360840394; duration=0sec 2023-04-01 14:54:00,462 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:00,482 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/9c4eea65a763491bae5cceb560a2ec92 2023-04-01 14:54:00,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:54:00,671 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:54:00,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360840698, max timestamp=1680360840720Num uniques 859; 2023-04-01 14:54:00,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360840698, max timestamp=1680360840720Num uniques 859; 2023-04-01 14:54:00,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:00,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:00,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360840698, max timestamp=1680360840720Num uniques 859; 2023-04-01 14:54:00,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:00,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680360840781, max timestamp=1680360840800Num uniques 1018; 2023-04-01 14:54:00,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 14:54:00,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680360840781, max timestamp=1680360840800Num uniques 1018; 2023-04-01 14:54:00,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 14:54:00,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680360840781, max timestamp=1680360840800Num uniques 1018; 2023-04-01 14:54:00,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 14:54:00,877 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/44b17b86272d46729a9c59f1ebb59eb3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/44b17b86272d46729a9c59f1ebb59eb3 2023-04-01 14:54:00,889 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/C of 8f567672fffd4bc77debf33ecd862db3 into 44b17b86272d46729a9c59f1ebb59eb3(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:00,889 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:00,889 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/C, priority=13, startTime=1680360840394; duration=0sec 2023-04-01 14:54:00,889 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:00,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:00,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360900890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,909 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/fc4946c2a600471bbe2f34348019d727 2023-04-01 14:54:00,938 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/5313bb79e45d404e8950b99ad924c92a 2023-04-01 14:54:00,951 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/9c4eea65a763491bae5cceb560a2ec92 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9c4eea65a763491bae5cceb560a2ec92 2023-04-01 14:54:00,958 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9c4eea65a763491bae5cceb560a2ec92, entries=150, sequenceid=601, filesize=11.9 K 2023-04-01 14:54:00,959 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/fc4946c2a600471bbe2f34348019d727 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/fc4946c2a600471bbe2f34348019d727 2023-04-01 14:54:00,972 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/fc4946c2a600471bbe2f34348019d727, entries=150, sequenceid=601, filesize=11.9 K 2023-04-01 14:54:00,974 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/5313bb79e45d404e8950b99ad924c92a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5313bb79e45d404e8950b99ad924c92a 2023-04-01 14:54:00,982 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5313bb79e45d404e8950b99ad924c92a, entries=150, sequenceid=601, filesize=11.9 K 2023-04-01 14:54:00,983 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 8f567672fffd4bc77debf33ecd862db3 in 534ms, sequenceid=601, compaction requested=false 2023-04-01 14:54:00,983 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:00,983 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:00,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:00,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:00,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:00,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:00,985 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:00,985 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:00,985 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,986 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@460f3593[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:00,989 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:00,989 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,991 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:00,991 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:00,991 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:00,991 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:00,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:00,992 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:00,993 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:00,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:00,996 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:00,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:00,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:00,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:00,996 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:00,999 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,999 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:00,999 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:01,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:01,000 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:01,003 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:54:01,004 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=147.60 KB heapSize=420 KB 2023-04-01 14:54:01,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:54:01,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 14:54:01,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:54:01,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 14:54:01,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:54:01,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 14:54:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,012 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,012 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:01,012 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:01,012 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:01,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:01,013 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:01,013 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:01,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:01,013 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:01,013 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:01,012 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:01,012 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:01,013 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:01,013 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:01,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:01,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:01,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/d6938a8f46364038aae733dbb2c9eeae 2023-04-01 14:54:01,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:01,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360901023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/9cb049cbf9cd4d34875550c1f2cad415 2023-04-01 14:54:01,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/777d4c86b15b4b08aab233f34a788f7a 2023-04-01 14:54:01,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/d6938a8f46364038aae733dbb2c9eeae as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d6938a8f46364038aae733dbb2c9eeae 2023-04-01 14:54:01,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d6938a8f46364038aae733dbb2c9eeae, entries=150, sequenceid=658, filesize=11.9 K 2023-04-01 14:54:01,091 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/9cb049cbf9cd4d34875550c1f2cad415 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/9cb049cbf9cd4d34875550c1f2cad415 2023-04-01 14:54:01,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/9cb049cbf9cd4d34875550c1f2cad415, entries=150, sequenceid=658, filesize=11.9 K 2023-04-01 14:54:01,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/777d4c86b15b4b08aab233f34a788f7a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/777d4c86b15b4b08aab233f34a788f7a 2023-04-01 14:54:01,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/777d4c86b15b4b08aab233f34a788f7a, entries=150, sequenceid=658, filesize=11.9 K 2023-04-01 14:54:01,102 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~419.95 KB/430032, currentSize=33.54 KB/34350 for 8f567672fffd4bc77debf33ecd862db3 in 97ms, sequenceid=658, compaction requested=true 2023-04-01 14:54:01,102 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:01,102 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:01,102 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:01,102 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:01,102 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:01,102 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:01,103 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:01,103 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/A is initiating minor compaction (all files) 2023-04-01 14:54:01,103 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:01,103 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/A in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:01,103 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/B is initiating minor compaction (all files) 2023-04-01 14:54:01,103 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/B in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:01,103 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9de69b3ef5d14c44b050b0b93efa28f8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9c4eea65a763491bae5cceb560a2ec92, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d6938a8f46364038aae733dbb2c9eeae] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=36.8 K 2023-04-01 14:54:01,103 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/535a160555d947968d734cfb99bbe38c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/fc4946c2a600471bbe2f34348019d727, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/9cb049cbf9cd4d34875550c1f2cad415] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=36.8 K 2023-04-01 14:54:01,104 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 9de69b3ef5d14c44b050b0b93efa28f8, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680360839182 2023-04-01 14:54:01,104 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 535a160555d947968d734cfb99bbe38c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680360839182 2023-04-01 14:54:01,104 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 9c4eea65a763491bae5cceb560a2ec92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680360839537 2023-04-01 14:54:01,104 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting fc4946c2a600471bbe2f34348019d727, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680360839537 2023-04-01 14:54:01,105 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d6938a8f46364038aae733dbb2c9eeae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680360840881 2023-04-01 14:54:01,105 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9cb049cbf9cd4d34875550c1f2cad415, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680360840881 2023-04-01 14:54:01,113 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:01,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:01,114 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#A#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:01,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:01,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:01,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:01,116 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:01,116 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:01,116 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:01,116 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:01,117 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,117 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:01,118 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#B#compaction#106 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:01,118 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:01,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:01,118 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,120 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:01,120 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:01,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:01,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:01,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:01,121 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:01,121 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:01,121 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:01,121 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:01,122 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:01,122 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:01,122 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:54:01,122 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-01 14:54:01,122 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:54:01,122 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:01,122 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:54:01,122 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:01,122 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:54:01,122 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:01,128 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/dd7c05fe35e94592b997cc2a12d2b97a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/dd7c05fe35e94592b997cc2a12d2b97a 2023-04-01 14:54:01,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:54:01,132 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:54:01,133 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/26517808d50040d4b4fefef536ef3d08 2023-04-01 14:54:01,134 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/693efc1ff8734131b4ed6438e8ca5bc3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/693efc1ff8734131b4ed6438e8ca5bc3 2023-04-01 14:54:01,135 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/A of 8f567672fffd4bc77debf33ecd862db3 into dd7c05fe35e94592b997cc2a12d2b97a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:01,135 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:01,135 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/A, priority=13, startTime=1680360841102; duration=0sec 2023-04-01 14:54:01,135 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:01,136 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:01,137 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:01,137 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/C is initiating minor compaction (all files) 2023-04-01 14:54:01,137 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/C in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:01,137 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/44b17b86272d46729a9c59f1ebb59eb3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5313bb79e45d404e8950b99ad924c92a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/777d4c86b15b4b08aab233f34a788f7a] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=36.8 K 2023-04-01 14:54:01,138 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 44b17b86272d46729a9c59f1ebb59eb3, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1680360839182 2023-04-01 14:54:01,139 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 5313bb79e45d404e8950b99ad924c92a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680360839537 2023-04-01 14:54:01,140 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 777d4c86b15b4b08aab233f34a788f7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680360840881 2023-04-01 14:54:01,141 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/B of 8f567672fffd4bc77debf33ecd862db3 into 693efc1ff8734131b4ed6438e8ca5bc3(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:01,141 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:01,141 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/B, priority=13, startTime=1680360841102; duration=0sec 2023-04-01 14:54:01,141 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:01,147 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#C#compaction#109 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:01,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360841158, max timestamp=1680360841170Num uniques 859; 2023-04-01 14:54:01,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360841158, max timestamp=1680360841170Num uniques 859; 2023-04-01 14:54:01,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360841158, max timestamp=1680360841170Num uniques 859; 2023-04-01 14:54:01,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:01,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:01,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:01,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680360841182, max timestamp=1680360841212Num uniques 1018; 2023-04-01 14:54:01,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680360841182, max timestamp=1680360841212Num uniques 1018; 2023-04-01 14:54:01,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-01 14:54:01,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-01 14:54:01,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680360841182, max timestamp=1680360841212Num uniques 1018; 2023-04-01 14:54:01,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-01 14:54:01,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:01,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360901251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:01,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360901355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,550 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/c5a536ee1e5a44f19ebc0c6737145556 2023-04-01 14:54:01,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:01,562 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/78944a39d4a643f2a6ff4db5b839846b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/78944a39d4a643f2a6ff4db5b839846b 2023-04-01 14:54:01,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360901560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,572 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/C of 8f567672fffd4bc77debf33ecd862db3 into 78944a39d4a643f2a6ff4db5b839846b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:01,573 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:01,573 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/C, priority=13, startTime=1680360841102; duration=0sec 2023-04-01 14:54:01,573 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:01,591 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/a12b729b1e644792946f1d9829d15f56 2023-04-01 14:54:01,597 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/26517808d50040d4b4fefef536ef3d08 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/26517808d50040d4b4fefef536ef3d08 2023-04-01 14:54:01,604 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/26517808d50040d4b4fefef536ef3d08, entries=150, sequenceid=666, filesize=11.9 K 2023-04-01 14:54:01,605 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/c5a536ee1e5a44f19ebc0c6737145556 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c5a536ee1e5a44f19ebc0c6737145556 2023-04-01 14:54:01,615 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c5a536ee1e5a44f19ebc0c6737145556, entries=150, sequenceid=666, filesize=11.9 K 2023-04-01 14:54:01,616 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/a12b729b1e644792946f1d9829d15f56 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a12b729b1e644792946f1d9829d15f56 2023-04-01 14:54:01,624 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a12b729b1e644792946f1d9829d15f56, entries=150, sequenceid=666, filesize=11.9 K 2023-04-01 14:54:01,625 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 8f567672fffd4bc77debf33ecd862db3 in 503ms, sequenceid=666, compaction requested=false 2023-04-01 14:54:01,625 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:01,625 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:01,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:01,631 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:01,631 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:01,633 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:01,634 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:01,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b0ac92e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:01,636 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:01,636 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:01,638 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:01,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:01,638 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:01,639 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,639 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:01,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:01,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:01,647 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:01,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:01,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:01,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:01,647 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:01,649 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,649 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,649 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:01,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:01,650 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,651 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:01,654 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:01,655 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:01,655 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:01,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:01,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:01,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:01,655 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:01,655 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:01,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:01,655 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:01,655 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:01,655 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:01,655 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,656 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:01,656 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:01,656 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:01,755 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:01,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:01,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:01,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:01,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:01,759 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:01,759 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:01,759 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:01,759 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:01,759 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,759 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:01,760 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:01,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:01,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:01,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:01,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:01,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:01,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:01,763 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:01,763 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:01,763 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:01,763 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:01,763 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:01,763 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:01,763 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:54:01,763 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-01 14:54:01,763 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:54:01,764 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:01,764 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:54:01,764 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:01,764 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:54:01,764 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:01,780 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/befdde198dd4475a9553a98cf56d1b73 2023-04-01 14:54:01,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:54:01,822 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:54:01,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:01,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360901826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:01,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360901827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:01,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51728 deadline: 1680360901835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,840 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:01,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:01,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360901867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360901930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:01,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360901930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:02,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360902136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:02,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360902136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,192 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/83e5be922e0d4f08aa8ebe0a5d9e97c1 2023-04-01 14:54:02,223 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/7b741563d40c4cf28ed913c401a7b3cb 2023-04-01 14:54:02,232 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/befdde198dd4475a9553a98cf56d1b73 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/befdde198dd4475a9553a98cf56d1b73 2023-04-01 14:54:02,244 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/befdde198dd4475a9553a98cf56d1b73, entries=150, sequenceid=723, filesize=11.9 K 2023-04-01 14:54:02,246 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/83e5be922e0d4f08aa8ebe0a5d9e97c1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/83e5be922e0d4f08aa8ebe0a5d9e97c1 2023-04-01 14:54:02,259 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/83e5be922e0d4f08aa8ebe0a5d9e97c1, entries=150, sequenceid=723, filesize=11.9 K 2023-04-01 14:54:02,261 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/7b741563d40c4cf28ed913c401a7b3cb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/7b741563d40c4cf28ed913c401a7b3cb 2023-04-01 14:54:02,269 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/7b741563d40c4cf28ed913c401a7b3cb, entries=150, sequenceid=723, filesize=11.9 K 2023-04-01 14:54:02,271 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=33.54 KB/34350 for 8f567672fffd4bc77debf33ecd862db3 in 508ms, sequenceid=723, compaction requested=true 2023-04-01 14:54:02,271 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:02,271 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:02,274 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,274 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:02,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cd38b6d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:02,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:02,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,281 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:02,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:02,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:02,283 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:02,283 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:02,283 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:02,284 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:02,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:02,287 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:02,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:02,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:02,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:02,287 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:02,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,294 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:02,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:02,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:02,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:02,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:02,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:02,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:02,303 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:02,303 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:02,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:02,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:54:02,379 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:54:02,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:54:02,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:02,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:54:02,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:02,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:54:02,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:02,403 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:02,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:02,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:02,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:02,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:02,409 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:02,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:02,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:02,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:02,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,410 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:02,411 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:02,411 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,411 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:02,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:02,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:02,413 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:02,413 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:02,413 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:02,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:54:02,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:54:02,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:02,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/76a2bf6bab58493aaf4268da81313997 2023-04-01 14:54:02,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:02,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2891f2a1[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:02,426 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:02,426 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,427 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:02,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:02,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:02,429 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,429 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:02,431 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:02,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:02,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360842409, max timestamp=1680360842433Num uniques 859; 2023-04-01 14:54:02,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:02,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360842409, max timestamp=1680360842433Num uniques 859; 2023-04-01 14:54:02,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:02,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360842409, max timestamp=1680360842433Num uniques 859; 2023-04-01 14:54:02,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:02,440 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:02,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:02,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:02,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:02,440 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:02,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,444 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:02,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:02,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:02,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:02,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:02,450 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:02,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:02,451 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,451 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:02,451 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:02,451 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:02,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:02,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/515e0b7d35724a258e01f13850bee58c 2023-04-01 14:54:02,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680360842458, max timestamp=1680360842478Num uniques 1018; 2023-04-01 14:54:02,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 14:54:02,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680360842458, max timestamp=1680360842478Num uniques 1018; 2023-04-01 14:54:02,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680360842458, max timestamp=1680360842478Num uniques 1018; 2023-04-01 14:54:02,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 14:54:02,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 14:54:02,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360902507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:02,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360902510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:02,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360902511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,551 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:02,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:02,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:02,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:02,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:02,564 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:02,564 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:02,564 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:02,564 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:02,565 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,565 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,568 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,568 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:02,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:02,568 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,568 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:02,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:02,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:02,572 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,573 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:54:02,573 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:54:02,573 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:02,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:02,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@170de98d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:02,578 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:02,578 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,580 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:02,581 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:02,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:02,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:02,582 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:02,582 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:02,582 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:02,582 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:02,582 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:02,582 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:02,582 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:02,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:02,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,585 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,597 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:02,597 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:02,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:02,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:02,597 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,598 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,598 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,598 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:02,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:02,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:02,598 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,598 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,598 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,598 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:02,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:02,598 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:02,598 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:02,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:02,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:02,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 430 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360902611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:02,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360902614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:02,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360902617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,698 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:02,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:02,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:02,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:02,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:02,702 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:02,702 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:02,702 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:02,702 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:02,702 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,702 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:02,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:02,704 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:02,704 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,704 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:02,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:02,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:02,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:02,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:02,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:02,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:02,705 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,706 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:54:02,706 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:54:02,706 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,706 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:02,706 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:02,706 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,706 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:02,706 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:02,707 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,708 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,708 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:02,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a5301fb[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:02,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:02,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,711 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:02,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:02,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:02,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:02,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,713 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,713 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:02,713 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:02,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:02,715 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:02,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:02,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:02,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:02,715 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:02,717 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,717 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,717 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,719 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:02,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:02,722 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:02,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:02,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:02,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:02,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:02,722 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,722 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,722 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,722 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:02,722 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:02,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:02,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:02,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 432 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360902815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:02,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360902818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:02,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360902819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,822 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:02,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:02,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:02,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:02,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:02,825 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:02,825 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:02,825 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:02,825 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:02,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,828 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:02,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:02,829 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,829 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:02,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:02,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:02,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:02,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:02,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:02,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:02,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:02,830 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,830 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:54:02,830 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:54:02,830 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:02,831 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:02,831 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,831 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:02,831 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:02,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,832 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,833 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,833 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:02,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a2b7b3f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:02,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:02,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,835 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:02,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:02,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:02,835 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,835 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:02,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:02,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:02,842 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:02,842 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:02,842 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:02,842 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:02,842 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:02,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:02,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:02,847 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,848 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,850 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,851 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:02,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:02,851 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,851 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:02,851 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,851 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:02,851 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:02,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:02,851 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,852 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:02,852 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:02,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:02,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:02,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:02,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:02,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:02,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/b8809e82af674b218946decb76a91c75 2023-04-01 14:54:02,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/76a2bf6bab58493aaf4268da81313997 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/76a2bf6bab58493aaf4268da81313997 2023-04-01 14:54:02,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/76a2bf6bab58493aaf4268da81313997, entries=250, sequenceid=733, filesize=16.7 K 2023-04-01 14:54:02,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/515e0b7d35724a258e01f13850bee58c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/515e0b7d35724a258e01f13850bee58c 2023-04-01 14:54:02,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/515e0b7d35724a258e01f13850bee58c, entries=150, sequenceid=733, filesize=11.9 K 2023-04-01 14:54:02,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/b8809e82af674b218946decb76a91c75 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b8809e82af674b218946decb76a91c75 2023-04-01 14:54:02,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b8809e82af674b218946decb76a91c75, entries=150, sequenceid=733, filesize=11.9 K 2023-04-01 14:54:02,926 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 8f567672fffd4bc77debf33ecd862db3 in 547ms, sequenceid=733, compaction requested=true 2023-04-01 14:54:02,926 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:02,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:02,927 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:54:02,927 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:54:02,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:02,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:02,930 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54816 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:54:02,930 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/A is initiating minor compaction (all files) 2023-04-01 14:54:02,930 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/A in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,930 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/dd7c05fe35e94592b997cc2a12d2b97a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/26517808d50040d4b4fefef536ef3d08, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/befdde198dd4475a9553a98cf56d1b73, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/76a2bf6bab58493aaf4268da81313997] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=53.5 K 2023-04-01 14:54:02,930 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49936 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:54:02,930 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/B is initiating minor compaction (all files) 2023-04-01 14:54:02,930 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/B in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,930 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/693efc1ff8734131b4ed6438e8ca5bc3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c5a536ee1e5a44f19ebc0c6737145556, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/83e5be922e0d4f08aa8ebe0a5d9e97c1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/515e0b7d35724a258e01f13850bee58c] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=48.8 K 2023-04-01 14:54:02,931 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting dd7c05fe35e94592b997cc2a12d2b97a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680360840881 2023-04-01 14:54:02,931 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 693efc1ff8734131b4ed6438e8ca5bc3, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680360840881 2023-04-01 14:54:02,931 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 26517808d50040d4b4fefef536ef3d08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680360841015 2023-04-01 14:54:02,931 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c5a536ee1e5a44f19ebc0c6737145556, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680360841015 2023-04-01 14:54:02,932 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting befdde198dd4475a9553a98cf56d1b73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1680360841241 2023-04-01 14:54:02,932 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 83e5be922e0d4f08aa8ebe0a5d9e97c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1680360841241 2023-04-01 14:54:02,933 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 76a2bf6bab58493aaf4268da81313997, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680360841821 2023-04-01 14:54:02,933 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 515e0b7d35724a258e01f13850bee58c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680360841821 2023-04-01 14:54:02,943 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#A#compaction#117 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:02,945 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#B#compaction#118 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:02,951 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:02,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:02,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:02,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:02,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:02,960 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:02,961 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:02,961 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:02,961 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:02,964 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,964 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,966 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,966 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:02,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:02,966 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:02,966 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:02,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:02,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:02,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:02,977 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:02,977 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:02,977 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:02,978 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:02,978 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:02,979 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:02,979 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. started... 2023-04-01 14:54:02,979 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-04-01 14:54:02,979 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:54:02,979 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:02,979 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:54:02,979 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:02,979 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:54:02,979 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:03,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:54:03,121 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. as already flushing 2023-04-01 14:54:03,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360903124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360903129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360903129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360903231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360903232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360903232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,384 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/3a7362a57c8442408283704815fc1424 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/3a7362a57c8442408283704815fc1424 2023-04-01 14:54:03,385 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/92af0315bad74868ae4f4887c0523905 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/92af0315bad74868ae4f4887c0523905 2023-04-01 14:54:03,391 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/A of 8f567672fffd4bc77debf33ecd862db3 into 3a7362a57c8442408283704815fc1424(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:03,391 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:03,391 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/A, priority=12, startTime=1680360842926; duration=0sec 2023-04-01 14:54:03,391 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/B of 8f567672fffd4bc77debf33ecd862db3 into 92af0315bad74868ae4f4887c0523905(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:03,391 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:03,391 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/B, priority=12, startTime=1680360842927; duration=0sec 2023-04-01 14:54:03,391 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:03,391 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:03,391 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:54:03,392 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49936 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:54:03,392 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/C is initiating minor compaction (all files) 2023-04-01 14:54:03,393 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/C in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:03,393 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/78944a39d4a643f2a6ff4db5b839846b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a12b729b1e644792946f1d9829d15f56, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/7b741563d40c4cf28ed913c401a7b3cb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b8809e82af674b218946decb76a91c75] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=48.8 K 2023-04-01 14:54:03,393 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 78944a39d4a643f2a6ff4db5b839846b, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1680360840881 2023-04-01 14:54:03,393 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting a12b729b1e644792946f1d9829d15f56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680360841015 2023-04-01 14:54:03,394 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 7b741563d40c4cf28ed913c401a7b3cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1680360841241 2023-04-01 14:54:03,394 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting b8809e82af674b218946decb76a91c75, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680360841821 2023-04-01 14:54:03,401 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/5219fac1e8a94b728fcf08f43eb76dab 2023-04-01 14:54:03,402 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#C#compaction#120 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:03,417 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/8566541ec50244db8d60e597591134ae as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/8566541ec50244db8d60e597591134ae 2023-04-01 14:54:03,422 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/C of 8f567672fffd4bc77debf33ecd862db3 into 8566541ec50244db8d60e597591134ae(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:03,422 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:03,422 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/C, priority=12, startTime=1680360842927; duration=0sec 2023-04-01 14:54:03,422 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:03,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360903434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360903435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360903435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51650 deadline: 1680360903738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51666 deadline: 1680360903738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:03,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51672 deadline: 1680360903740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:03,794 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6aad2e18 to 127.0.0.1:60465 2023-04-01 14:54:03,794 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49a45b6d to 127.0.0.1:60465 2023-04-01 14:54:03,794 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:03,794 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:03,795 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fef7a21 to 127.0.0.1:60465 2023-04-01 14:54:03,795 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:03,800 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd3abbd to 127.0.0.1:60465 2023-04-01 14:54:03,800 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:03,814 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/741e4fe5a226458db6d1dc7f80422dfe 2023-04-01 14:54:04,224 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/0b0a3b3bd79a4f14bd6b5af0fb468c99 2023-04-01 14:54:04,229 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/5219fac1e8a94b728fcf08f43eb76dab as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/5219fac1e8a94b728fcf08f43eb76dab 2023-04-01 14:54:04,233 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/5219fac1e8a94b728fcf08f43eb76dab, entries=150, sequenceid=787, filesize=11.9 K 2023-04-01 14:54:04,233 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/741e4fe5a226458db6d1dc7f80422dfe as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/741e4fe5a226458db6d1dc7f80422dfe 2023-04-01 14:54:04,237 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/741e4fe5a226458db6d1dc7f80422dfe, entries=150, sequenceid=787, filesize=11.9 K 2023-04-01 14:54:04,238 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/0b0a3b3bd79a4f14bd6b5af0fb468c99 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/0b0a3b3bd79a4f14bd6b5af0fb468c99 2023-04-01 14:54:04,241 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/0b0a3b3bd79a4f14bd6b5af0fb468c99, entries=150, sequenceid=787, filesize=11.9 K 2023-04-01 14:54:04,242 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.33 KB/408912, currentSize=40.25 KB/41220 for 8f567672fffd4bc77debf33ecd862db3 in 1263ms, sequenceid=787, compaction requested=false 2023-04-01 14:54:04,242 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:04,242 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:04,242 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:04,242 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:04,242 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,243 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:04,243 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:04,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:54:04,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-01 14:54:04,244 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fea27db to 127.0.0.1:60465 2023-04-01 14:54:04,244 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:04,244 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56c8e45e to 127.0.0.1:60465 2023-04-01 14:54:04,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:54:04,244 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61e72c30 to 127.0.0.1:60465 2023-04-01 14:54:04,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:04,244 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:04,244 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:04,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:54:04,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:04,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:54:04,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:04,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:04,244 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:04,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:04,247 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:04,247 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:04,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:04,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d088a21[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:04,248 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:04,248 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:04,250 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:04,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:04,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:04,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:04,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:04,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:04,250 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,250 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:04,252 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:04,252 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:04,252 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,252 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:04,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:04,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:04,254 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:04,254 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:04,254 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:04,254 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:04,254 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:04,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:04,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:04,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:04,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:04,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:04,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:04,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:04,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:04,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:04,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:04,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:04,258 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,260 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:04,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:04,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:04,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:04,265 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:04,265 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:04,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:04,265 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:04,365 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:04,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:04,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=802 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/c424a07d555c4564b97865cd3e42b31a 2023-04-01 14:54:05,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=802 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/6c47eeffad7c4d3eaa8aa419cf1c5369 2023-04-01 14:54:05,218 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6038a026 to 127.0.0.1:60465 2023-04-01 14:54:05,218 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:05,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=802 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/9658ca9f214d407585e462cff8d7eff9 2023-04-01 14:54:05,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/c424a07d555c4564b97865cd3e42b31a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c424a07d555c4564b97865cd3e42b31a 2023-04-01 14:54:05,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c424a07d555c4564b97865cd3e42b31a, entries=150, sequenceid=802, filesize=11.9 K 2023-04-01 14:54:05,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/6c47eeffad7c4d3eaa8aa419cf1c5369 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6c47eeffad7c4d3eaa8aa419cf1c5369 2023-04-01 14:54:05,487 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6c47eeffad7c4d3eaa8aa419cf1c5369, entries=150, sequenceid=802, filesize=11.9 K 2023-04-01 14:54:05,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/9658ca9f214d407585e462cff8d7eff9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9658ca9f214d407585e462cff8d7eff9 2023-04-01 14:54:05,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9658ca9f214d407585e462cff8d7eff9, entries=150, sequenceid=802, filesize=11.9 K 2023-04-01 14:54:05,492 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=6.71 KB/6870 for 8f567672fffd4bc77debf33ecd862db3 in 1248ms, sequenceid=802, compaction requested=true 2023-04-01 14:54:05,492 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:05,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:05,493 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:05,493 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:05,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:05,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:05,494 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:05,494 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/A is initiating minor compaction (all files) 2023-04-01 14:54:05,494 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:05,494 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/A in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:05,494 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/B is initiating minor compaction (all files) 2023-04-01 14:54:05,494 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/3a7362a57c8442408283704815fc1424, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/5219fac1e8a94b728fcf08f43eb76dab, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c424a07d555c4564b97865cd3e42b31a] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=37.0 K 2023-04-01 14:54:05,494 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/B in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:05,494 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/92af0315bad74868ae4f4887c0523905, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/741e4fe5a226458db6d1dc7f80422dfe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6c47eeffad7c4d3eaa8aa419cf1c5369] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=37.0 K 2023-04-01 14:54:05,494 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 3a7362a57c8442408283704815fc1424, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680360841821 2023-04-01 14:54:05,495 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 92af0315bad74868ae4f4887c0523905, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680360841821 2023-04-01 14:54:05,495 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 5219fac1e8a94b728fcf08f43eb76dab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1680360842504 2023-04-01 14:54:05,495 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 741e4fe5a226458db6d1dc7f80422dfe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1680360842504 2023-04-01 14:54:05,495 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting c424a07d555c4564b97865cd3e42b31a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1680360843128 2023-04-01 14:54:05,495 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6c47eeffad7c4d3eaa8aa419cf1c5369, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1680360843128 2023-04-01 14:54:05,503 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#A#compaction#126 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:05,505 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#B#compaction#127 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:05,849 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63859c62 to 127.0.0.1:60465 2023-04-01 14:54:05,849 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:05,849 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 14:54:05,849 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 104 2023-04-01 14:54:05,849 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 299 2023-04-01 14:54:05,849 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 125 2023-04-01 14:54:05,849 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 46 2023-04-01 14:54:05,849 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 103 2023-04-01 14:54:05,849 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 14:54:05,850 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4030 2023-04-01 14:54:05,850 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 3926 2023-04-01 14:54:05,850 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 14:54:05,850 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 1767 2023-04-01 14:54:05,850 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 5297 rows 2023-04-01 14:54:05,850 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 1778 2023-04-01 14:54:05,850 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 5332 rows 2023-04-01 14:54:05,850 INFO [Listener at localhost/46667] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 14:54:05,850 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44ef43e3 to 127.0.0.1:60465 2023-04-01 14:54:05,850 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:05,854 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 14:54:05,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-01 14:54:05,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:05,872 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360845872"}]},"ts":"1680360845872"} 2023-04-01 14:54:05,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 14:54:05,873 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 14:54:05,874 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 14:54:05,876 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8f567672fffd4bc77debf33ecd862db3, UNASSIGN}] 2023-04-01 14:54:05,877 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8f567672fffd4bc77debf33ecd862db3, UNASSIGN 2023-04-01 14:54:05,878 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=8f567672fffd4bc77debf33ecd862db3, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:05,878 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360845878"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360845878"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360845878"}]},"ts":"1680360845878"} 2023-04-01 14:54:05,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:54:05,914 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/464e7e241c3f4fabb57c47eb6e9f21b5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/464e7e241c3f4fabb57c47eb6e9f21b5 2023-04-01 14:54:05,915 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/daf4e97fd0694ecb99f855c7a0f1f311 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/daf4e97fd0694ecb99f855c7a0f1f311 2023-04-01 14:54:05,919 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/A of 8f567672fffd4bc77debf33ecd862db3 into 464e7e241c3f4fabb57c47eb6e9f21b5(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:05,919 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:05,919 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/A, priority=13, startTime=1680360845492; duration=0sec 2023-04-01 14:54:05,919 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:05,920 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:05,920 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/B of 8f567672fffd4bc77debf33ecd862db3 into daf4e97fd0694ecb99f855c7a0f1f311(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:05,920 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:05,920 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/B, priority=13, startTime=1680360845493; duration=0sec 2023-04-01 14:54:05,920 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:05,921 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:05,921 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 8f567672fffd4bc77debf33ecd862db3/C is initiating minor compaction (all files) 2023-04-01 14:54:05,921 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8f567672fffd4bc77debf33ecd862db3/C in TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:05,921 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/8566541ec50244db8d60e597591134ae, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/0b0a3b3bd79a4f14bd6b5af0fb468c99, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9658ca9f214d407585e462cff8d7eff9] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp, totalSize=37.0 K 2023-04-01 14:54:05,921 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 8566541ec50244db8d60e597591134ae, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1680360841821 2023-04-01 14:54:05,922 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 0b0a3b3bd79a4f14bd6b5af0fb468c99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1680360842504 2023-04-01 14:54:05,922 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 9658ca9f214d407585e462cff8d7eff9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1680360843128 2023-04-01 14:54:05,929 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8f567672fffd4bc77debf33ecd862db3#C#compaction#128 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:05,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 14:54:06,038 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:54:06,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8f567672fffd4bc77debf33ecd862db3, disabling compactions & flushes 2023-04-01 14:54:06,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions to complete for region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:06,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 14:54:06,338 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/d2418cc9b2d84c9c80c12b93674a1b0b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d2418cc9b2d84c9c80c12b93674a1b0b 2023-04-01 14:54:06,343 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8f567672fffd4bc77debf33ecd862db3/C of 8f567672fffd4bc77debf33ecd862db3 into d2418cc9b2d84c9c80c12b93674a1b0b(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:06,343 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:06,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:06,343 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3., storeName=8f567672fffd4bc77debf33ecd862db3/C, priority=13, startTime=1680360845493; duration=0sec 2023-04-01 14:54:06,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:06,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. after waiting 0 ms 2023-04-01 14:54:06,343 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:06,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:06,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8f567672fffd4bc77debf33ecd862db3 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-01 14:54:06,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=A 2023-04-01 14:54:06,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:06,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=B 2023-04-01 14:54:06,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:06,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8f567672fffd4bc77debf33ecd862db3, store=C 2023-04-01 14:54:06,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:06,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 14:54:06,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=810 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/78dee4f9f68c4605bd5af062ef27ddc9 2023-04-01 14:54:06,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 14:54:07,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=810 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/fe6b46cf760d48cda959b0a69f48f7fb 2023-04-01 14:54:07,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=810 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/77c0792cf7c747c5ad929199e20f26c1 2023-04-01 14:54:07,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/A/78dee4f9f68c4605bd5af062ef27ddc9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/78dee4f9f68c4605bd5af062ef27ddc9 2023-04-01 14:54:07,583 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/78dee4f9f68c4605bd5af062ef27ddc9, entries=50, sequenceid=810, filesize=7.1 K 2023-04-01 14:54:07,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/B/fe6b46cf760d48cda959b0a69f48f7fb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/fe6b46cf760d48cda959b0a69f48f7fb 2023-04-01 14:54:07,589 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/fe6b46cf760d48cda959b0a69f48f7fb, entries=50, sequenceid=810, filesize=7.1 K 2023-04-01 14:54:07,590 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/.tmp/C/77c0792cf7c747c5ad929199e20f26c1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/77c0792cf7c747c5ad929199e20f26c1 2023-04-01 14:54:07,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/77c0792cf7c747c5ad929199e20f26c1, entries=50, sequenceid=810, filesize=7.1 K 2023-04-01 14:54:07,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 8f567672fffd4bc77debf33ecd862db3 in 1253ms, sequenceid=810, compaction requested=false 2023-04-01 14:54:07,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/13ac35b3a6b840688c4e8d8353c3ec67, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61edd2acaed64d9bb367e6f6edecb1e5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bad19c79e21d4ce28b5926803ef14b13, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/2ab6282b23f34da488dba7de696a6003, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/4cb570775fb04655b4b3d25cb16e2117, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9641c33393e8466094455e59b19d0434, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/158a1470a5f943f5b6f934cca449ec3d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/70cabfde3a0f4c8e8dc26009cb8dab57, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bb36332297f14c51abd836053a6c1e38, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/aa7ae60660674470a740fa8d973fe9c5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/52e51d8b5e89470e87a50be53f622d24, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a87792459884c50b01956c9e052ffe2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/0f78789ccabf4ab1a7afb236e11f28bc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/debaf99cd651438a93aac38bcd809beb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/aa14893980714543a540926875608171, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b1f5c00b84854934a2c1a8db83ca0bbc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a1540b62abd43549b542e01dbf1d61e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/01b9266a0a644957bf05a6d0b4092dc7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c2fa08a8410942399fef5331f0af4271, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/cc8bfc12d6064eb58a865cf1061ae2cd, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bebac4fd62e34219be52f933b4813b82, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d711be7a983b4064a495cc9cab471606, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b534248419ca4774844b16b3a4af06fb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/66340225efbf4da68f8135fff0669f91, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b6dc69b449fb4b1baaa443fe3d1acbb7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b74b834f3110443f831d09787e470ab7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/2ccd7d2081d243c1b706e8b6b1f1ff29, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c8ec1d442a9e4a9b92c70ba6dc3caae7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61ddf7447bc44042a84bb255ac5f2e1a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/21a99ba58e234bdca518931cc36f3c6c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d5e68b4c76224df68d8296f145434961, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9de69b3ef5d14c44b050b0b93efa28f8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/0ca488973b3741299d3002bd2e649dde, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9c4eea65a763491bae5cceb560a2ec92, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/dd7c05fe35e94592b997cc2a12d2b97a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d6938a8f46364038aae733dbb2c9eeae, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/26517808d50040d4b4fefef536ef3d08, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/befdde198dd4475a9553a98cf56d1b73, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/76a2bf6bab58493aaf4268da81313997, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/3a7362a57c8442408283704815fc1424, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/5219fac1e8a94b728fcf08f43eb76dab, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c424a07d555c4564b97865cd3e42b31a] to archive 2023-04-01 14:54:07,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:54:07,625 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/13ac35b3a6b840688c4e8d8353c3ec67 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/13ac35b3a6b840688c4e8d8353c3ec67 2023-04-01 14:54:07,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61edd2acaed64d9bb367e6f6edecb1e5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61edd2acaed64d9bb367e6f6edecb1e5 2023-04-01 14:54:07,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bad19c79e21d4ce28b5926803ef14b13 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bad19c79e21d4ce28b5926803ef14b13 2023-04-01 14:54:07,630 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/2ab6282b23f34da488dba7de696a6003 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/2ab6282b23f34da488dba7de696a6003 2023-04-01 14:54:07,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/4cb570775fb04655b4b3d25cb16e2117 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/4cb570775fb04655b4b3d25cb16e2117 2023-04-01 14:54:07,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9641c33393e8466094455e59b19d0434 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9641c33393e8466094455e59b19d0434 2023-04-01 14:54:07,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/158a1470a5f943f5b6f934cca449ec3d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/158a1470a5f943f5b6f934cca449ec3d 2023-04-01 14:54:07,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/70cabfde3a0f4c8e8dc26009cb8dab57 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/70cabfde3a0f4c8e8dc26009cb8dab57 2023-04-01 14:54:07,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bb36332297f14c51abd836053a6c1e38 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bb36332297f14c51abd836053a6c1e38 2023-04-01 14:54:07,637 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/aa7ae60660674470a740fa8d973fe9c5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/aa7ae60660674470a740fa8d973fe9c5 2023-04-01 14:54:07,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/52e51d8b5e89470e87a50be53f622d24 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/52e51d8b5e89470e87a50be53f622d24 2023-04-01 14:54:07,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a87792459884c50b01956c9e052ffe2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a87792459884c50b01956c9e052ffe2 2023-04-01 14:54:07,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/0f78789ccabf4ab1a7afb236e11f28bc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/0f78789ccabf4ab1a7afb236e11f28bc 2023-04-01 14:54:07,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/debaf99cd651438a93aac38bcd809beb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/debaf99cd651438a93aac38bcd809beb 2023-04-01 14:54:07,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/aa14893980714543a540926875608171 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/aa14893980714543a540926875608171 2023-04-01 14:54:07,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b1f5c00b84854934a2c1a8db83ca0bbc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b1f5c00b84854934a2c1a8db83ca0bbc 2023-04-01 14:54:07,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a1540b62abd43549b542e01dbf1d61e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/7a1540b62abd43549b542e01dbf1d61e 2023-04-01 14:54:07,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/01b9266a0a644957bf05a6d0b4092dc7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/01b9266a0a644957bf05a6d0b4092dc7 2023-04-01 14:54:07,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c2fa08a8410942399fef5331f0af4271 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c2fa08a8410942399fef5331f0af4271 2023-04-01 14:54:07,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/cc8bfc12d6064eb58a865cf1061ae2cd to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/cc8bfc12d6064eb58a865cf1061ae2cd 2023-04-01 14:54:07,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bebac4fd62e34219be52f933b4813b82 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/bebac4fd62e34219be52f933b4813b82 2023-04-01 14:54:07,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d711be7a983b4064a495cc9cab471606 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d711be7a983b4064a495cc9cab471606 2023-04-01 14:54:07,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b534248419ca4774844b16b3a4af06fb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b534248419ca4774844b16b3a4af06fb 2023-04-01 14:54:07,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/66340225efbf4da68f8135fff0669f91 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/66340225efbf4da68f8135fff0669f91 2023-04-01 14:54:07,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b6dc69b449fb4b1baaa443fe3d1acbb7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b6dc69b449fb4b1baaa443fe3d1acbb7 2023-04-01 14:54:07,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b74b834f3110443f831d09787e470ab7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/b74b834f3110443f831d09787e470ab7 2023-04-01 14:54:07,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/2ccd7d2081d243c1b706e8b6b1f1ff29 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/2ccd7d2081d243c1b706e8b6b1f1ff29 2023-04-01 14:54:07,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c8ec1d442a9e4a9b92c70ba6dc3caae7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c8ec1d442a9e4a9b92c70ba6dc3caae7 2023-04-01 14:54:07,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61ddf7447bc44042a84bb255ac5f2e1a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/61ddf7447bc44042a84bb255ac5f2e1a 2023-04-01 14:54:07,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/21a99ba58e234bdca518931cc36f3c6c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/21a99ba58e234bdca518931cc36f3c6c 2023-04-01 14:54:07,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d5e68b4c76224df68d8296f145434961 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d5e68b4c76224df68d8296f145434961 2023-04-01 14:54:07,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9de69b3ef5d14c44b050b0b93efa28f8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9de69b3ef5d14c44b050b0b93efa28f8 2023-04-01 14:54:07,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/0ca488973b3741299d3002bd2e649dde to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/0ca488973b3741299d3002bd2e649dde 2023-04-01 14:54:07,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9c4eea65a763491bae5cceb560a2ec92 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/9c4eea65a763491bae5cceb560a2ec92 2023-04-01 14:54:07,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/dd7c05fe35e94592b997cc2a12d2b97a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/dd7c05fe35e94592b997cc2a12d2b97a 2023-04-01 14:54:07,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d6938a8f46364038aae733dbb2c9eeae to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/d6938a8f46364038aae733dbb2c9eeae 2023-04-01 14:54:07,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/26517808d50040d4b4fefef536ef3d08 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/26517808d50040d4b4fefef536ef3d08 2023-04-01 14:54:07,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/befdde198dd4475a9553a98cf56d1b73 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/befdde198dd4475a9553a98cf56d1b73 2023-04-01 14:54:07,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/76a2bf6bab58493aaf4268da81313997 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/76a2bf6bab58493aaf4268da81313997 2023-04-01 14:54:07,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/3a7362a57c8442408283704815fc1424 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/3a7362a57c8442408283704815fc1424 2023-04-01 14:54:07,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/5219fac1e8a94b728fcf08f43eb76dab to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/5219fac1e8a94b728fcf08f43eb76dab 2023-04-01 14:54:07,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c424a07d555c4564b97865cd3e42b31a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/c424a07d555c4564b97865cd3e42b31a 2023-04-01 14:54:07,696 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/d75019da49664d0b9f8ef7d7cd9bad92, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/35a6fd30b8dd4a4581d7a9b3f94b2424, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/34211a328a124e9d99ba7fa32af71aba, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/48ff1771ac384a14bf543a872b3fbf1b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/342e06fa898140018c1f1eac36448297, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6913081a186443f68cc3649a50f4c3b3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/74794d9a379e4928a5a4d5566fb82e64, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5711a6e25e664fb3a5867f3e3faff1a8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/15e37978d1ae4066be77dc7846cdc3a3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/a33c5689d4b44fc8936bec04c7d86b30, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/40adb6ee151f42a2bba2cbb7b47225d9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e350ed8b7b4c4b598f299eb9a395adbe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c3d7e9c3fa5a46fa8952eb3ec14b1e2d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4b4224dbdb84f4eaca648dfc91e188a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/0626ae85cb8b42cb96d8dbf5075ef3d4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5fd703e9f9db479db7715eabaf23c7ee, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/be7ce9140753472693346e70dd95b94c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/429ccab98ddc48abbff9ae5c828e5af8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dbaf61f4e3b9431eb0aed46b2fef3713, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4a5c024c2884fa7a47fcb86c0e99d06, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6bb2939464d149aeab53b692ec54c9b0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5c2d89246c37449da664a9493df1d2a7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/cef7763c72344ec4b9d7886ac908ebff, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/1f327c36c28148179c6834ee02e18ad8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/34035bc684cd47da915ddf9d267a9ecb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/2e217420341d44bab2ad99c62b5b5d38, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/400fa5b54b8544bc9f016ad5b883ccaa, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dec371d528074c74995edf109f43d3f1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e8a19b568f4048e8ae0f6b00a48af166, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/ade23a6539324592aedd63279a3d5063, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/11d07ac7d28c499694359cafb774947c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/535a160555d947968d734cfb99bbe38c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e6db14164d324e4fa90dbe547d44ea40, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/fc4946c2a600471bbe2f34348019d727, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/693efc1ff8734131b4ed6438e8ca5bc3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/9cb049cbf9cd4d34875550c1f2cad415, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c5a536ee1e5a44f19ebc0c6737145556, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/83e5be922e0d4f08aa8ebe0a5d9e97c1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/92af0315bad74868ae4f4887c0523905, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/515e0b7d35724a258e01f13850bee58c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/741e4fe5a226458db6d1dc7f80422dfe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6c47eeffad7c4d3eaa8aa419cf1c5369] to archive 2023-04-01 14:54:07,697 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:54:07,699 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/d75019da49664d0b9f8ef7d7cd9bad92 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/d75019da49664d0b9f8ef7d7cd9bad92 2023-04-01 14:54:07,700 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/35a6fd30b8dd4a4581d7a9b3f94b2424 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/35a6fd30b8dd4a4581d7a9b3f94b2424 2023-04-01 14:54:07,701 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/34211a328a124e9d99ba7fa32af71aba to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/34211a328a124e9d99ba7fa32af71aba 2023-04-01 14:54:07,702 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/48ff1771ac384a14bf543a872b3fbf1b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/48ff1771ac384a14bf543a872b3fbf1b 2023-04-01 14:54:07,703 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/342e06fa898140018c1f1eac36448297 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/342e06fa898140018c1f1eac36448297 2023-04-01 14:54:07,705 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6913081a186443f68cc3649a50f4c3b3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6913081a186443f68cc3649a50f4c3b3 2023-04-01 14:54:07,706 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/74794d9a379e4928a5a4d5566fb82e64 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/74794d9a379e4928a5a4d5566fb82e64 2023-04-01 14:54:07,707 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5711a6e25e664fb3a5867f3e3faff1a8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5711a6e25e664fb3a5867f3e3faff1a8 2023-04-01 14:54:07,708 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/15e37978d1ae4066be77dc7846cdc3a3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/15e37978d1ae4066be77dc7846cdc3a3 2023-04-01 14:54:07,709 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/a33c5689d4b44fc8936bec04c7d86b30 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/a33c5689d4b44fc8936bec04c7d86b30 2023-04-01 14:54:07,710 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/40adb6ee151f42a2bba2cbb7b47225d9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/40adb6ee151f42a2bba2cbb7b47225d9 2023-04-01 14:54:07,711 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e350ed8b7b4c4b598f299eb9a395adbe to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e350ed8b7b4c4b598f299eb9a395adbe 2023-04-01 14:54:07,712 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c3d7e9c3fa5a46fa8952eb3ec14b1e2d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c3d7e9c3fa5a46fa8952eb3ec14b1e2d 2023-04-01 14:54:07,713 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4b4224dbdb84f4eaca648dfc91e188a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4b4224dbdb84f4eaca648dfc91e188a 2023-04-01 14:54:07,714 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/0626ae85cb8b42cb96d8dbf5075ef3d4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/0626ae85cb8b42cb96d8dbf5075ef3d4 2023-04-01 14:54:07,716 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5fd703e9f9db479db7715eabaf23c7ee to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5fd703e9f9db479db7715eabaf23c7ee 2023-04-01 14:54:07,717 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/be7ce9140753472693346e70dd95b94c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/be7ce9140753472693346e70dd95b94c 2023-04-01 14:54:07,718 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/429ccab98ddc48abbff9ae5c828e5af8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/429ccab98ddc48abbff9ae5c828e5af8 2023-04-01 14:54:07,719 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dbaf61f4e3b9431eb0aed46b2fef3713 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dbaf61f4e3b9431eb0aed46b2fef3713 2023-04-01 14:54:07,720 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4a5c024c2884fa7a47fcb86c0e99d06 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c4a5c024c2884fa7a47fcb86c0e99d06 2023-04-01 14:54:07,721 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6bb2939464d149aeab53b692ec54c9b0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6bb2939464d149aeab53b692ec54c9b0 2023-04-01 14:54:07,722 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5c2d89246c37449da664a9493df1d2a7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/5c2d89246c37449da664a9493df1d2a7 2023-04-01 14:54:07,723 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/cef7763c72344ec4b9d7886ac908ebff to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/cef7763c72344ec4b9d7886ac908ebff 2023-04-01 14:54:07,725 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/1f327c36c28148179c6834ee02e18ad8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/1f327c36c28148179c6834ee02e18ad8 2023-04-01 14:54:07,726 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/34035bc684cd47da915ddf9d267a9ecb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/34035bc684cd47da915ddf9d267a9ecb 2023-04-01 14:54:07,727 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/2e217420341d44bab2ad99c62b5b5d38 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/2e217420341d44bab2ad99c62b5b5d38 2023-04-01 14:54:07,728 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/400fa5b54b8544bc9f016ad5b883ccaa to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/400fa5b54b8544bc9f016ad5b883ccaa 2023-04-01 14:54:07,729 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dec371d528074c74995edf109f43d3f1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/dec371d528074c74995edf109f43d3f1 2023-04-01 14:54:07,731 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e8a19b568f4048e8ae0f6b00a48af166 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e8a19b568f4048e8ae0f6b00a48af166 2023-04-01 14:54:07,732 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/ade23a6539324592aedd63279a3d5063 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/ade23a6539324592aedd63279a3d5063 2023-04-01 14:54:07,733 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/11d07ac7d28c499694359cafb774947c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/11d07ac7d28c499694359cafb774947c 2023-04-01 14:54:07,734 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/535a160555d947968d734cfb99bbe38c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/535a160555d947968d734cfb99bbe38c 2023-04-01 14:54:07,735 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e6db14164d324e4fa90dbe547d44ea40 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/e6db14164d324e4fa90dbe547d44ea40 2023-04-01 14:54:07,736 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/fc4946c2a600471bbe2f34348019d727 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/fc4946c2a600471bbe2f34348019d727 2023-04-01 14:54:07,737 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/693efc1ff8734131b4ed6438e8ca5bc3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/693efc1ff8734131b4ed6438e8ca5bc3 2023-04-01 14:54:07,738 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/9cb049cbf9cd4d34875550c1f2cad415 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/9cb049cbf9cd4d34875550c1f2cad415 2023-04-01 14:54:07,739 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c5a536ee1e5a44f19ebc0c6737145556 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/c5a536ee1e5a44f19ebc0c6737145556 2023-04-01 14:54:07,740 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/83e5be922e0d4f08aa8ebe0a5d9e97c1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/83e5be922e0d4f08aa8ebe0a5d9e97c1 2023-04-01 14:54:07,742 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/92af0315bad74868ae4f4887c0523905 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/92af0315bad74868ae4f4887c0523905 2023-04-01 14:54:07,743 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/515e0b7d35724a258e01f13850bee58c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/515e0b7d35724a258e01f13850bee58c 2023-04-01 14:54:07,743 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/741e4fe5a226458db6d1dc7f80422dfe to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/741e4fe5a226458db6d1dc7f80422dfe 2023-04-01 14:54:07,745 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6c47eeffad7c4d3eaa8aa419cf1c5369 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/6c47eeffad7c4d3eaa8aa419cf1c5369 2023-04-01 14:54:07,748 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b9aed4cc7bd2485c9bfc119415ae933c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9482ec81074c45bea68019193d5c6994, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/707ee18b5131436e880a0f0a2428f9f5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cbca22a408f74e998217a59eb9ee28f6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/e467c815dace4c0197416ee3458efbed, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/33854bc7845b466b914d098f9b697875, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9fe19587c61a4546b016db370bbaf865, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/39885dd7027447c6aa05cc854bc871ef, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b1a47fa0a1d2421fa4bf2de0692a29be, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5d95db6bf513463f85c658f3bea4a907, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cc7f7adf311c4ce8adca0593fcd48ea4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b9415ec1edf041a18fdf98e847694f76, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/8ecdc287ba7b40c3859a979ff13d3cb5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/695bb9ee66524f7a8da3d89071909d49, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/58cb89e666f8437191a8ce43c81b97cc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/2a11458b76ba4bf6bfcf61ebc13f5598, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20acbd7f759c497da886476a17c4df19, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/c61e51d14246431a82a481f86edc4f8e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/574052d55a6846df816207b8f429706a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9be951fda7c44d1db8edae5c47fb1090, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a8402536d4304bad8220e101ef3b5200, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/ac460c0327ed43d3ac92260eb419d69f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20266795213245918556a040ca78a4b0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b8ea6a682fd846c6b3de10239b856071, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d11690b05bfd476e9ecf01c4aaa1bb70, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/f6e9185711ac45578d039c8fc3b3c769, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/1aa88a34713945858701e04e42560881, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d6cfe478fe454af3893e5793d939f7af, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9398eabd8d68406ea8dbe7b9e70c6fad, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/60ed58535b604a9e9bdc25bf0949a679, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/fba2cdcc921c4f60ab5256cdc0d2b128, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/44b17b86272d46729a9c59f1ebb59eb3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/40469b401d9f4782a55ef7fba7fc8d66, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5313bb79e45d404e8950b99ad924c92a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/78944a39d4a643f2a6ff4db5b839846b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/777d4c86b15b4b08aab233f34a788f7a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a12b729b1e644792946f1d9829d15f56, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/7b741563d40c4cf28ed913c401a7b3cb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/8566541ec50244db8d60e597591134ae, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b8809e82af674b218946decb76a91c75, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/0b0a3b3bd79a4f14bd6b5af0fb468c99, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9658ca9f214d407585e462cff8d7eff9] to archive 2023-04-01 14:54:07,749 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:54:07,750 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b9aed4cc7bd2485c9bfc119415ae933c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b9aed4cc7bd2485c9bfc119415ae933c 2023-04-01 14:54:07,751 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9482ec81074c45bea68019193d5c6994 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9482ec81074c45bea68019193d5c6994 2023-04-01 14:54:07,752 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/707ee18b5131436e880a0f0a2428f9f5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/707ee18b5131436e880a0f0a2428f9f5 2023-04-01 14:54:07,753 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cbca22a408f74e998217a59eb9ee28f6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cbca22a408f74e998217a59eb9ee28f6 2023-04-01 14:54:07,754 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/e467c815dace4c0197416ee3458efbed to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/e467c815dace4c0197416ee3458efbed 2023-04-01 14:54:07,755 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/33854bc7845b466b914d098f9b697875 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/33854bc7845b466b914d098f9b697875 2023-04-01 14:54:07,755 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9fe19587c61a4546b016db370bbaf865 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9fe19587c61a4546b016db370bbaf865 2023-04-01 14:54:07,757 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/39885dd7027447c6aa05cc854bc871ef to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/39885dd7027447c6aa05cc854bc871ef 2023-04-01 14:54:07,757 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b1a47fa0a1d2421fa4bf2de0692a29be to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b1a47fa0a1d2421fa4bf2de0692a29be 2023-04-01 14:54:07,758 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5d95db6bf513463f85c658f3bea4a907 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5d95db6bf513463f85c658f3bea4a907 2023-04-01 14:54:07,759 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cc7f7adf311c4ce8adca0593fcd48ea4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/cc7f7adf311c4ce8adca0593fcd48ea4 2023-04-01 14:54:07,761 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b9415ec1edf041a18fdf98e847694f76 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b9415ec1edf041a18fdf98e847694f76 2023-04-01 14:54:07,762 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/8ecdc287ba7b40c3859a979ff13d3cb5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/8ecdc287ba7b40c3859a979ff13d3cb5 2023-04-01 14:54:07,763 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/695bb9ee66524f7a8da3d89071909d49 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/695bb9ee66524f7a8da3d89071909d49 2023-04-01 14:54:07,764 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/58cb89e666f8437191a8ce43c81b97cc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/58cb89e666f8437191a8ce43c81b97cc 2023-04-01 14:54:07,765 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/2a11458b76ba4bf6bfcf61ebc13f5598 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/2a11458b76ba4bf6bfcf61ebc13f5598 2023-04-01 14:54:07,766 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20acbd7f759c497da886476a17c4df19 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20acbd7f759c497da886476a17c4df19 2023-04-01 14:54:07,767 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/c61e51d14246431a82a481f86edc4f8e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/c61e51d14246431a82a481f86edc4f8e 2023-04-01 14:54:07,768 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/574052d55a6846df816207b8f429706a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/574052d55a6846df816207b8f429706a 2023-04-01 14:54:07,769 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9be951fda7c44d1db8edae5c47fb1090 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9be951fda7c44d1db8edae5c47fb1090 2023-04-01 14:54:07,770 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a8402536d4304bad8220e101ef3b5200 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a8402536d4304bad8220e101ef3b5200 2023-04-01 14:54:07,771 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/ac460c0327ed43d3ac92260eb419d69f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/ac460c0327ed43d3ac92260eb419d69f 2023-04-01 14:54:07,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20266795213245918556a040ca78a4b0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/20266795213245918556a040ca78a4b0 2023-04-01 14:54:07,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b8ea6a682fd846c6b3de10239b856071 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b8ea6a682fd846c6b3de10239b856071 2023-04-01 14:54:07,774 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d11690b05bfd476e9ecf01c4aaa1bb70 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d11690b05bfd476e9ecf01c4aaa1bb70 2023-04-01 14:54:07,775 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/f6e9185711ac45578d039c8fc3b3c769 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/f6e9185711ac45578d039c8fc3b3c769 2023-04-01 14:54:07,776 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/1aa88a34713945858701e04e42560881 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/1aa88a34713945858701e04e42560881 2023-04-01 14:54:07,777 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d6cfe478fe454af3893e5793d939f7af to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d6cfe478fe454af3893e5793d939f7af 2023-04-01 14:54:07,778 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9398eabd8d68406ea8dbe7b9e70c6fad to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9398eabd8d68406ea8dbe7b9e70c6fad 2023-04-01 14:54:07,779 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/60ed58535b604a9e9bdc25bf0949a679 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/60ed58535b604a9e9bdc25bf0949a679 2023-04-01 14:54:07,780 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/fba2cdcc921c4f60ab5256cdc0d2b128 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/fba2cdcc921c4f60ab5256cdc0d2b128 2023-04-01 14:54:07,781 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/44b17b86272d46729a9c59f1ebb59eb3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/44b17b86272d46729a9c59f1ebb59eb3 2023-04-01 14:54:07,782 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/40469b401d9f4782a55ef7fba7fc8d66 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/40469b401d9f4782a55ef7fba7fc8d66 2023-04-01 14:54:07,783 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5313bb79e45d404e8950b99ad924c92a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/5313bb79e45d404e8950b99ad924c92a 2023-04-01 14:54:07,784 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/78944a39d4a643f2a6ff4db5b839846b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/78944a39d4a643f2a6ff4db5b839846b 2023-04-01 14:54:07,786 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/777d4c86b15b4b08aab233f34a788f7a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/777d4c86b15b4b08aab233f34a788f7a 2023-04-01 14:54:07,787 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a12b729b1e644792946f1d9829d15f56 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/a12b729b1e644792946f1d9829d15f56 2023-04-01 14:54:07,788 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/7b741563d40c4cf28ed913c401a7b3cb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/7b741563d40c4cf28ed913c401a7b3cb 2023-04-01 14:54:07,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/8566541ec50244db8d60e597591134ae to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/8566541ec50244db8d60e597591134ae 2023-04-01 14:54:07,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b8809e82af674b218946decb76a91c75 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/b8809e82af674b218946decb76a91c75 2023-04-01 14:54:07,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/0b0a3b3bd79a4f14bd6b5af0fb468c99 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/0b0a3b3bd79a4f14bd6b5af0fb468c99 2023-04-01 14:54:07,793 DEBUG [StoreCloser-TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9658ca9f214d407585e462cff8d7eff9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/9658ca9f214d407585e462cff8d7eff9 2023-04-01 14:54:07,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/recovered.edits/813.seqid, newMaxSeqId=813, maxSeqId=1 2023-04-01 14:54:07,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3. 2023-04-01 14:54:07,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8f567672fffd4bc77debf33ecd862db3: 2023-04-01 14:54:07,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:54:07,801 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=8f567672fffd4bc77debf33ecd862db3, regionState=CLOSED 2023-04-01 14:54:07,802 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360847801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360847801"}]},"ts":"1680360847801"} 2023-04-01 14:54:07,804 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-01 14:54:07,804 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 8f567672fffd4bc77debf33ecd862db3, server=jenkins-hbase4.apache.org,40863,1680360818567 in 1.9240 sec 2023-04-01 14:54:07,806 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-01 14:54:07,806 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8f567672fffd4bc77debf33ecd862db3, UNASSIGN in 1.9280 sec 2023-04-01 14:54:07,806 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360847806"}]},"ts":"1680360847806"} 2023-04-01 14:54:07,807 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 14:54:07,809 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 14:54:07,811 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9480 sec 2023-04-01 14:54:07,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-01 14:54:07,977 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-01 14:54:07,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-01 14:54:07,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:07,990 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:07,991 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:07,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-01 14:54:07,996 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:54:07,999 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/recovered.edits] 2023-04-01 14:54:08,003 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/464e7e241c3f4fabb57c47eb6e9f21b5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/464e7e241c3f4fabb57c47eb6e9f21b5 2023-04-01 14:54:08,005 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/78dee4f9f68c4605bd5af062ef27ddc9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/A/78dee4f9f68c4605bd5af062ef27ddc9 2023-04-01 14:54:08,008 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/daf4e97fd0694ecb99f855c7a0f1f311 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/daf4e97fd0694ecb99f855c7a0f1f311 2023-04-01 14:54:08,010 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/fe6b46cf760d48cda959b0a69f48f7fb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/B/fe6b46cf760d48cda959b0a69f48f7fb 2023-04-01 14:54:08,012 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/77c0792cf7c747c5ad929199e20f26c1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/77c0792cf7c747c5ad929199e20f26c1 2023-04-01 14:54:08,014 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d2418cc9b2d84c9c80c12b93674a1b0b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/C/d2418cc9b2d84c9c80c12b93674a1b0b 2023-04-01 14:54:08,016 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/recovered.edits/813.seqid to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3/recovered.edits/813.seqid 2023-04-01 14:54:08,017 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/8f567672fffd4bc77debf33ecd862db3 2023-04-01 14:54:08,017 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:54:08,020 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:08,041 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 14:54:08,083 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 14:54:08,085 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:08,085 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 14:54:08,085 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680360848085"}]},"ts":"9223372036854775807"} 2023-04-01 14:54:08,087 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 14:54:08,087 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8f567672fffd4bc77debf33ecd862db3, NAME => 'TestAcidGuarantees,,1680360822501.8f567672fffd4bc77debf33ecd862db3.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 14:54:08,087 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 14:54:08,087 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680360848087"}]},"ts":"9223372036854775807"} 2023-04-01 14:54:08,089 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 14:54:08,092 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:08,093 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 109 msec 2023-04-01 14:54:08,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-01 14:54:08,093 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-01 14:54:08,109 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=237 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f5d57d6-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:40863-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-620753973_15 at /127.0.0.1:40726 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-620753973_15 at /127.0.0.1:40868 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-620753973_15 at /127.0.0.1:40776 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-620753973_15 at /127.0.0.1:40804 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f5d57d6-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f5d57d6-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f5d57d6-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=586 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 261) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 170), AvailableMemoryMB=3564 (was 4150) 2023-04-01 14:54:08,121 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=237, OpenFileDescriptor=586, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=169, AvailableMemoryMB=3564 2023-04-01 14:54:08,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 14:54:08,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 14:54:08,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:08,125 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 14:54:08,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-01 14:54:08,126 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 14:54:08,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 14:54:08,127 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:08,128 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f empty. 2023-04-01 14:54:08,128 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:08,128 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:54:08,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 14:54:08,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 14:54:08,538 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 14:54:08,541 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9a2eee92d9536149526dfe0ebd08db5f, NAME => 'TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp 2023-04-01 14:54:08,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 14:54:08,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:54:08,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9a2eee92d9536149526dfe0ebd08db5f, disabling compactions & flushes 2023-04-01 14:54:08,947 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:08,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:08,948 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. after waiting 0 ms 2023-04-01 14:54:08,948 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:08,948 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:08,948 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:08,950 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 14:54:08,951 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360848951"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360848951"}]},"ts":"1680360848951"} 2023-04-01 14:54:08,952 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 14:54:08,953 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 14:54:08,953 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360848953"}]},"ts":"1680360848953"} 2023-04-01 14:54:08,954 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 14:54:08,957 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2eee92d9536149526dfe0ebd08db5f, ASSIGN}] 2023-04-01 14:54:08,959 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2eee92d9536149526dfe0ebd08db5f, ASSIGN 2023-04-01 14:54:08,959 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2eee92d9536149526dfe0ebd08db5f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40863,1680360818567; forceNewPlan=false, retain=false 2023-04-01 14:54:09,111 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=9a2eee92d9536149526dfe0ebd08db5f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:09,111 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360849110"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360849110"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360849110"}]},"ts":"1680360849110"} 2023-04-01 14:54:09,112 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:54:09,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 14:54:09,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:09,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9a2eee92d9536149526dfe0ebd08db5f, NAME => 'TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:54:09,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:09,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:54:09,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:09,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:09,270 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:09,271 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A 2023-04-01 14:54:09,271 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A 2023-04-01 14:54:09,272 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:54:09,272 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a2eee92d9536149526dfe0ebd08db5f columnFamilyName A 2023-04-01 14:54:09,273 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(310): Store=9a2eee92d9536149526dfe0ebd08db5f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:54:09,273 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:09,274 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B 2023-04-01 14:54:09,274 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B 2023-04-01 14:54:09,274 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:54:09,274 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a2eee92d9536149526dfe0ebd08db5f columnFamilyName B 2023-04-01 14:54:09,275 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(310): Store=9a2eee92d9536149526dfe0ebd08db5f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:54:09,275 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:09,276 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C 2023-04-01 14:54:09,276 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C 2023-04-01 14:54:09,276 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:54:09,277 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a2eee92d9536149526dfe0ebd08db5f columnFamilyName C 2023-04-01 14:54:09,278 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(310): Store=9a2eee92d9536149526dfe0ebd08db5f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:54:09,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:09,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:09,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:09,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 14:54:09,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:09,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 14:54:09,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9a2eee92d9536149526dfe0ebd08db5f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70654212, jitterRate=0.0528298020362854}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 14:54:09,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:09,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., pid=18, masterSystemTime=1680360849264 2023-04-01 14:54:09,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:09,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:09,289 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=9a2eee92d9536149526dfe0ebd08db5f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:09,289 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360849289"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680360849289"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680360849289"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680360849289"}]},"ts":"1680360849289"} 2023-04-01 14:54:09,293 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-01 14:54:09,293 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 in 178 msec 2023-04-01 14:54:09,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-01 14:54:09,295 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2eee92d9536149526dfe0ebd08db5f, ASSIGN in 336 msec 2023-04-01 14:54:09,295 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 14:54:09,295 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360849295"}]},"ts":"1680360849295"} 2023-04-01 14:54:09,296 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 14:54:09,299 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 14:54:09,301 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1760 sec 2023-04-01 14:54:10,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-01 14:54:10,231 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-01 14:54:10,232 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x7829b43e to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:10,237 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@186ae08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:10,238 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:10,242 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48974, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:10,244 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 14:54:10,246 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 14:54:10,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 14:54:10,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 14:54:10,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:10,278 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 14:54:10,279 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 14:54:10,279 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 14:54:10,284 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-01 14:54:10,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2eee92d9536149526dfe0ebd08db5f, REOPEN/MOVE}] 2023-04-01 14:54:10,300 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2eee92d9536149526dfe0ebd08db5f, REOPEN/MOVE 2023-04-01 14:54:10,301 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9a2eee92d9536149526dfe0ebd08db5f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,301 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360850300"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360850300"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360850300"}]},"ts":"1680360850300"} 2023-04-01 14:54:10,302 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:54:10,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9a2eee92d9536149526dfe0ebd08db5f, disabling compactions & flushes 2023-04-01 14:54:10,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:10,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:10,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. after waiting 0 ms 2023-04-01 14:54:10,456 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:10,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-01 14:54:10,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:10,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:10,460 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 9a2eee92d9536149526dfe0ebd08db5f to self. 2023-04-01 14:54:10,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,462 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9a2eee92d9536149526dfe0ebd08db5f, regionState=CLOSED 2023-04-01 14:54:10,462 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360850462"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360850462"}]},"ts":"1680360850462"} 2023-04-01 14:54:10,465 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-01 14:54:10,465 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 in 161 msec 2023-04-01 14:54:10,466 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2eee92d9536149526dfe0ebd08db5f, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40863,1680360818567; forceNewPlan=false, retain=true 2023-04-01 14:54:10,616 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9a2eee92d9536149526dfe0ebd08db5f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,617 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360850616"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360850616"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360850616"}]},"ts":"1680360850616"} 2023-04-01 14:54:10,618 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:54:10,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:10,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9a2eee92d9536149526dfe0ebd08db5f, NAME => 'TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:54:10,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:54:10,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,777 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,778 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A 2023-04-01 14:54:10,778 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A 2023-04-01 14:54:10,778 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:54:10,781 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a2eee92d9536149526dfe0ebd08db5f columnFamilyName A 2023-04-01 14:54:10,783 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(310): Store=9a2eee92d9536149526dfe0ebd08db5f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:54:10,783 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,784 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B 2023-04-01 14:54:10,784 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B 2023-04-01 14:54:10,785 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:54:10,785 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a2eee92d9536149526dfe0ebd08db5f columnFamilyName B 2023-04-01 14:54:10,786 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(310): Store=9a2eee92d9536149526dfe0ebd08db5f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:54:10,786 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,786 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C 2023-04-01 14:54:10,786 DEBUG [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C 2023-04-01 14:54:10,787 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:54:10,787 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a2eee92d9536149526dfe0ebd08db5f columnFamilyName C 2023-04-01 14:54:10,788 INFO [StoreOpener-9a2eee92d9536149526dfe0ebd08db5f-1] regionserver.HStore(310): Store=9a2eee92d9536149526dfe0ebd08db5f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:54:10,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:10,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 14:54:10,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9a2eee92d9536149526dfe0ebd08db5f; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73315677, jitterRate=0.09248872101306915}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 14:54:10,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:10,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., pid=23, masterSystemTime=1680360850770 2023-04-01 14:54:10,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:10,796 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:10,796 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9a2eee92d9536149526dfe0ebd08db5f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,796 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360850796"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680360850796"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680360850796"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680360850796"}]},"ts":"1680360850796"} 2023-04-01 14:54:10,799 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-01 14:54:10,799 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 in 179 msec 2023-04-01 14:54:10,800 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-01 14:54:10,800 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2eee92d9536149526dfe0ebd08db5f, REOPEN/MOVE in 500 msec 2023-04-01 14:54:10,802 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-01 14:54:10,803 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 518 msec 2023-04-01 14:54:10,805 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 546 msec 2023-04-01 14:54:10,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-01 14:54:10,815 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x21281a37 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:10,823 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55cbb115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:10,824 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:10,827 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:10,827 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:10,832 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:10,833 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:10,837 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:10,838 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:10,841 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:10,841 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:10,846 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:10,846 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:10,851 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:10,851 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:10,855 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:10,856 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x2efdb02f to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:10,859 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@92bed09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:10,866 DEBUG [hconnection-0x44daf48-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:10,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:10,867 DEBUG [hconnection-0x173fb09b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:10,867 DEBUG [hconnection-0x69e1869-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:10,867 DEBUG [hconnection-0x23daec1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:10,867 DEBUG [hconnection-0x66e641c0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:10,868 DEBUG [hconnection-0x3bf29349-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:10,869 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48976, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:10,869 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49000, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:10,869 DEBUG [hconnection-0xfa5edc1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:10,872 DEBUG [hconnection-0x5a7db0dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:10,873 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48986, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:10,873 DEBUG [hconnection-0x67cc265d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:10,873 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49014, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:10,873 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48994, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:10,875 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49020, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:10,877 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:10,877 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49034, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:10,877 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:10,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:10,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:10,916 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:10,916 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:10,916 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:10,916 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:10,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:54:10,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:10,918 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:10,918 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:10,920 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:10,920 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:10,920 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:10,920 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:10,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:10,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:10,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:10,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:10,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:10,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:10,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:10,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:10,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:10,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:10,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:10,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:10,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360910945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:10,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360910946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:10,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:10,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360910948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360910948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:10,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360910948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:10,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:10,972 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:10,972 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:10,972 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:10,980 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:10,980 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:10,980 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:10,980 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:10,980 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:10,980 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:10,980 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:10,980 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:10,980 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,984 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:10,984 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:10,987 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,987 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:10,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:10,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:10,988 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:10,988 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:10,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:10,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25b1418c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:10,989 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:10,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:10,991 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:10,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:10,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:10,991 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:10,991 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:10,991 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:10,992 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:10,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:10,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:10,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:10,996 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:10,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:10,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:10,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:10,996 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:10,998 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:10,998 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:10,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:10,998 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:10,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,001 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,002 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,006 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,006 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,006 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,006 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,006 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:11,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:11,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:11,006 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:11,006 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:11,006 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,007 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,007 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,007 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,007 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360911055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360911057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360911057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360911057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360911057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,107 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:11,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:11,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:11,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:11,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:11,115 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:11,115 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,116 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:11,116 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:11,116 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,116 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,117 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,118 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,118 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,118 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:11,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:11,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:11,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:11,120 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:11,120 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,120 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:11,128 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:11,128 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:11,128 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,128 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:11,128 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:11,128 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,128 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:11,128 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:11,128 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:11,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:11,133 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,134 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,134 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:11,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37ffd018[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:11,139 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:11,139 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,140 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,141 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,141 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:11,141 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:11,141 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:11,141 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:11,144 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:11,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:11,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:11,148 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:11,148 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:11,148 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:11,148 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:11,148 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:11,150 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,150 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,150 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,151 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,153 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,156 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:11,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:11,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:11,156 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:11,156 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:11,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,257 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:11,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:11,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:11,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360911259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360911259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360911260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360911261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360911262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:11,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:11,268 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:11,268 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,268 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:11,268 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:11,272 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,272 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,275 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,276 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,276 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:11,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:11,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:11,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:11,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:11,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:11,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:11,296 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:11,296 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:11,296 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,296 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:11,296 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:11,296 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,297 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:11,297 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:11,297 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,297 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:11,297 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:11,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,298 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,299 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,299 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:11,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:11,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@162cb75b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:11,301 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:11,301 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,303 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:11,303 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:11,303 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:11,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:11,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:11,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:11,311 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:11,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:11,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:11,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:11,311 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:11,317 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,317 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,317 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,325 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,340 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,340 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:11,340 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,340 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:11,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:11,340 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,340 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,341 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,341 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,341 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,341 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:11,341 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:11,416 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040181207421cd6e4528bfd2377ba5d28e03 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040181207421cd6e4528bfd2377ba5d28e03 2023-04-01 14:54:11,441 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:11,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:11,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:11,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:11,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:11,454 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:11,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:11,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:11,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,455 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,456 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,456 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:11,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:11,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:11,460 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:11,460 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,460 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:11,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:11,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:11,461 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,461 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:11,461 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:11,461 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:11,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:11,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:11,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:11,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,463 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,463 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:11,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:11,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f6c7e3d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:11,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,467 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:11,467 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:11,467 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:11,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:11,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:11,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:11,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:11,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:11,472 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:11,472 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:11,472 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:11,472 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:11,472 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:11,474 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,474 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,474 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,476 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,477 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,480 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,480 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,480 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:11,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:11,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:11,480 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,480 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,480 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,481 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,480 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,480 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:11,481 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:11,481 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360911565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360911575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,581 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:11,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360911581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360911581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:11,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360911581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:11,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:11,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:11,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:11,586 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:11,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:11,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:11,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,589 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,589 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:11,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:11,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:11,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:11,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:11,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:11,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:11,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:11,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:11,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,595 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:11,595 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:11,595 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:11,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:11,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:11,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:11,598 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,598 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,599 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,599 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:11,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:11,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20fef37d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:11,600 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:11,600 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,601 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,601 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:11,602 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,602 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:11,602 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:11,603 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:11,605 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:11,605 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,605 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,605 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:11,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:11,609 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:11,609 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:11,609 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:11,609 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:11,609 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:11,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,612 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,617 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,624 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,624 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,625 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:11,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:11,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,625 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:11,625 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:11,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,725 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:11,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:11,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:11,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:11,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:11,730 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:11,730 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,730 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:11,730 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:11,730 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,730 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:11,736 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:11,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:11,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:11,738 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:11,738 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,738 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:11,738 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:11,738 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:11,738 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,738 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:11,738 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:11,738 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,738 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:11,739 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:11,739 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,739 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:11,739 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:11,740 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,740 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,740 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,740 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:11,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c8886a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:11,742 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:11,742 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,743 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:11,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:11,743 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,744 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:11,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,746 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:11,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:11,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:11,750 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:11,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:11,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:11,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:11,750 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:11,751 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,751 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,752 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,753 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,754 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,756 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,757 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,757 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,757 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,757 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:11,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:11,757 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,757 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:11,757 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,757 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,757 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:11,757 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,757 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:11,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,825 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/266a372cef5b4350ace50fef50662e99 2023-04-01 14:54:11,843 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/ac5563f060ba4f3599097004cc3c78cb 2023-04-01 14:54:11,857 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:11,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:11,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:11,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:11,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:11,861 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:11,861 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,861 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:11,861 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:11,862 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,862 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,863 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,864 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,864 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:11,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:11,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:11,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:11,865 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,865 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:11,865 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:11,865 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:11,866 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:11,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,868 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,868 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,868 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:11,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:11,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aad6c67[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:11,870 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:11,870 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,871 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,871 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:11,871 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:11,871 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:11,871 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,871 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:11,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:11,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,872 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:11,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:11,875 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:11,875 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:11,875 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:11,875 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:11,875 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:11,877 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,877 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,877 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:11,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,878 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:11,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,879 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:11,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,883 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,883 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,883 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,883 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:11,883 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:11,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:11,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,883 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,883 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,883 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:11,883 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,883 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:11,883 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:11,883 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:11,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:11,984 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:11,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:11,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:11,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:11,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:11,988 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:11,988 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,988 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:11,988 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:11,989 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,989 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:11,991 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:11,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:11,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:11,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:11,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:11,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:11,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:11,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:11,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:11,996 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:11,996 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:11,996 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,997 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:11,997 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:11,997 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:11,997 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:11,997 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:11,997 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:11,997 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:11,997 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:12,000 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:12,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76dc6499[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:12,002 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:12,002 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,003 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,003 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:12,003 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:12,003 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:12,003 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,004 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:12,006 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,006 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:12,006 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,006 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:12,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:12,010 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:12,010 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:12,010 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:12,010 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:12,010 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:12,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:12,019 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,021 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,025 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,025 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,025 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:12,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:12,025 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:12,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:12,025 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,025 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,025 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,025 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:12,026 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:12,025 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:12,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:12,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360912069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:12,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360912079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:12,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360912083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:12,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360912086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:12,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360912087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,126 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:12,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:12,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:12,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:12,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:12,130 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:12,130 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:12,130 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:12,130 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:12,131 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,131 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,133 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,133 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:12,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:12,133 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,133 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:12,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:12,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:12,136 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:12,136 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:12,136 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:12,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:12,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:12,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:12,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:12,149 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:12,153 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:12,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21ee0424[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:12,156 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:12,156 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,159 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:12,159 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,159 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:12,160 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,160 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,160 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,161 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:12,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:12,163 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:12,164 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:12,164 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:12,164 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:12,164 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:12,165 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,165 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,165 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,167 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,170 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:12,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:12,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:12,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:12,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:12,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:12,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:12,173 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:12,173 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:12,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/326c2c194b7d45ed8a32a2d522b15cea 2023-04-01 14:54:12,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/266a372cef5b4350ace50fef50662e99 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/266a372cef5b4350ace50fef50662e99 2023-04-01 14:54:12,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/266a372cef5b4350ace50fef50662e99, entries=200, sequenceid=17, filesize=31.9 K 2023-04-01 14:54:12,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/ac5563f060ba4f3599097004cc3c78cb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/ac5563f060ba4f3599097004cc3c78cb 2023-04-01 14:54:12,273 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:12,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:12,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:12,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:12,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:12,278 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:12,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:12,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:12,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:12,279 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,279 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,281 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,281 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:12,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:12,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:12,282 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,282 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:12,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:12,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/ac5563f060ba4f3599097004cc3c78cb, entries=150, sequenceid=17, filesize=11.6 K 2023-04-01 14:54:12,284 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/326c2c194b7d45ed8a32a2d522b15cea as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/326c2c194b7d45ed8a32a2d522b15cea 2023-04-01 14:54:12,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/326c2c194b7d45ed8a32a2d522b15cea, entries=150, sequenceid=17, filesize=11.6 K 2023-04-01 14:54:12,293 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 9a2eee92d9536149526dfe0ebd08db5f in 1384ms, sequenceid=17, compaction requested=false 2023-04-01 14:54:12,293 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:12,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:12,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:12,296 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:12,296 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:12,296 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:12,302 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:12,302 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:12,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:12,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:12,302 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 14:54:12,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:12,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:12,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:12,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:12,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:12,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:12,343 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401b603db06c9184b2b8a0c87d74eccaa88 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b603db06c9184b2b8a0c87d74eccaa88 2023-04-01 14:54:12,367 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/3620e2be25b6424ba0cb1266bca5e984 2023-04-01 14:54:12,398 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/cbbb03ffc372435f99d95e396a9eb0e4 2023-04-01 14:54:12,412 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/f44a229727614242b547b35565e2eafc 2023-04-01 14:54:12,422 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/3620e2be25b6424ba0cb1266bca5e984 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3620e2be25b6424ba0cb1266bca5e984 2023-04-01 14:54:12,429 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3620e2be25b6424ba0cb1266bca5e984, entries=150, sequenceid=37, filesize=25.1 K 2023-04-01 14:54:12,430 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/cbbb03ffc372435f99d95e396a9eb0e4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cbbb03ffc372435f99d95e396a9eb0e4 2023-04-01 14:54:12,435 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cbbb03ffc372435f99d95e396a9eb0e4, entries=150, sequenceid=37, filesize=11.6 K 2023-04-01 14:54:12,436 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/f44a229727614242b547b35565e2eafc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f44a229727614242b547b35565e2eafc 2023-04-01 14:54:12,442 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f44a229727614242b547b35565e2eafc, entries=150, sequenceid=37, filesize=11.6 K 2023-04-01 14:54:12,443 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 9a2eee92d9536149526dfe0ebd08db5f in 140ms, sequenceid=37, compaction requested=false 2023-04-01 14:54:12,443 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:12,443 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:12,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:12,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:12,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:12,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:12,444 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,445 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,445 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:12,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc843ab[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:12,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:12,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,448 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:12,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:12,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:12,448 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:12,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,455 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,456 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,456 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:12,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:12,459 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:12,459 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:12,459 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:12,459 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:12,459 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:12,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:12,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:12,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:12,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:12,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:12,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:12,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:12,469 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:12,469 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:12,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,569 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:12,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:12,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:12,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:12,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:12,574 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:12,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:12,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:12,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:12,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,577 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:12,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:12,577 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,578 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,578 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:12,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:12,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:12,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:12,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:12,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:12,589 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:12,589 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:12,589 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:12,589 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:12,590 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:12,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,592 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:12,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d77d801[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:12,593 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:12,594 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,595 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:12,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:12,596 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,596 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:12,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:12,597 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:12,601 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:12,601 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:12,601 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:12,601 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:12,601 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:12,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,605 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,607 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,611 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,613 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,613 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,613 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,613 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,613 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,613 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,613 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,613 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:12,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:12,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:12,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:12,614 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:12,613 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:12,613 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,614 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:12,733 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:12,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:12,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:12,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:12,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:12,737 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:12,737 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:12,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:12,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:12,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,740 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:12,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:12,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:12,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:12,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:12,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:12,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:12,747 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:12,747 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:12,747 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:12,747 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:12,747 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:12,747 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:12,747 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:12,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:12,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:12,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:12,749 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,749 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,750 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,750 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:12,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@689a839a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:12,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:12,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:12,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:12,753 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,753 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:12,758 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:12,758 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,758 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:12,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:12,761 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:12,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:12,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:12,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:12,761 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:12,762 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,764 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,768 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:12,768 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:12,768 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,768 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,768 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,768 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,768 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:12,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:12,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:12,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:12,768 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,769 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:12,769 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:12,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,869 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:12,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:12,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:12,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:12,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:12,874 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:12,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:12,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:12,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:12,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,876 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:12,876 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:12,877 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,877 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:12,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:12,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:12,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:12,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:12,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:12,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:12,901 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:12,901 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:12,901 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:12,901 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:12,901 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:12,901 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:12,901 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,901 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:12,901 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:12,903 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,903 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,904 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,904 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:12,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39846589[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:12,906 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:12,906 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,907 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,907 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:12,907 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:12,907 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:12,907 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,907 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:12,908 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:12,908 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,909 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:12,909 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:12,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:12,911 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:12,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:12,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:12,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:12,912 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:12,914 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,914 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,914 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:12,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:12,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:12,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,916 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,919 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:12,919 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:12,919 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,919 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,919 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:12,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:12,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:12,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:12,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:12,919 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:12,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:12,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:12,920 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:12,920 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:12,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:12,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:12,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:13,020 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:13,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:13,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:13,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:13,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:13,025 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:13,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:13,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:13,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:13,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,026 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,027 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:13,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:13,027 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:13,027 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,027 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:13,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:13,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:13,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:13,029 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:13,029 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:13,029 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:13,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:13,042 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:13,042 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:13,043 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:13,043 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:13,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:13,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:13,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:13,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:13,045 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:13,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:13,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:13,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@771fbc09[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:13,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:13,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,050 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:13,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:13,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:13,050 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,050 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:13,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:13,055 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:13,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:13,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:13,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:13,055 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:13,057 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,057 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:13,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,057 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:13,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:13,058 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,059 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:13,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:13,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:13,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:13,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:13,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:13,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:13,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,063 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,063 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,062 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:13,063 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:13,063 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:13,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:13,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:13,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:54:13,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:13,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:13,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:13,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:13,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:13,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:13,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360913144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360913146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360853136, max timestamp=1680360853148Num uniques 859; 2023-04-01 14:54:13,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360853136, max timestamp=1680360853148Num uniques 859; 2023-04-01 14:54:13,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:54:13,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:54:13,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360853136, max timestamp=1680360853147Num uniques 859; 2023-04-01 14:54:13,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:54:13,163 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:13,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:13,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:13,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360913169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,173 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401013f2e556a564d93afa54cb4852afb88 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401013f2e556a564d93afa54cb4852afb88 2023-04-01 14:54:13,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:13,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:13,174 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:13,174 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:13,174 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:13,174 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:13,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360913171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360913173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,177 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:13,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:13,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:13,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:13,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:13,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:13,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:13,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:13,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:13,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:13,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:13,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:13,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:13,198 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/869835375fbb458c9c53e2768a0b206a 2023-04-01 14:54:13,206 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,206 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:13,207 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,207 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@346bc53b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:13,211 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:13,211 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,219 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,219 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:13,219 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:13,219 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:13,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:13,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/f927000734d14f1aa61656211fdb6023 2023-04-01 14:54:13,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,221 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:13,221 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:13,221 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:13,226 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:13,226 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:13,226 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:13,226 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:13,226 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:13,235 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,235 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:13,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,236 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:13,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:13,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:13,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:13,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:13,237 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,239 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,242 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:13,242 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,242 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:13,242 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:13,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:13,242 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,242 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,242 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,242 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:13,242 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:13,242 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:13,242 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:13,242 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:13,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:13,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:13,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:13,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360913252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360913252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/02134c5d9a6a444f9072dd227ddaab50 2023-04-01 14:54:13,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/869835375fbb458c9c53e2768a0b206a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/869835375fbb458c9c53e2768a0b206a 2023-04-01 14:54:13,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360913274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360913276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360913279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/869835375fbb458c9c53e2768a0b206a, entries=250, sequenceid=52, filesize=38.7 K 2023-04-01 14:54:13,284 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/f927000734d14f1aa61656211fdb6023 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/f927000734d14f1aa61656211fdb6023 2023-04-01 14:54:13,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/f927000734d14f1aa61656211fdb6023, entries=150, sequenceid=52, filesize=11.6 K 2023-04-01 14:54:13,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/02134c5d9a6a444f9072dd227ddaab50 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/02134c5d9a6a444f9072dd227ddaab50 2023-04-01 14:54:13,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/02134c5d9a6a444f9072dd227ddaab50, entries=150, sequenceid=52, filesize=11.6 K 2023-04-01 14:54:13,299 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9a2eee92d9536149526dfe0ebd08db5f in 175ms, sequenceid=52, compaction requested=true 2023-04-01 14:54:13,299 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:13,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:13,300 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:13,300 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:13,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:13,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:13,303 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98062 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:13,303 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 9a2eee92d9536149526dfe0ebd08db5f/A is initiating minor compaction (all files) 2023-04-01 14:54:13,303 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2eee92d9536149526dfe0ebd08db5f/A in TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:13,303 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/266a372cef5b4350ace50fef50662e99, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3620e2be25b6424ba0cb1266bca5e984, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/869835375fbb458c9c53e2768a0b206a] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp, totalSize=95.8 K 2023-04-01 14:54:13,304 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:13,304 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 9a2eee92d9536149526dfe0ebd08db5f/B is initiating minor compaction (all files) 2023-04-01 14:54:13,304 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2eee92d9536149526dfe0ebd08db5f/B in TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:13,304 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/ac5563f060ba4f3599097004cc3c78cb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cbbb03ffc372435f99d95e396a9eb0e4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/f927000734d14f1aa61656211fdb6023] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp, totalSize=34.8 K 2023-04-01 14:54:13,305 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ac5563f060ba4f3599097004cc3c78cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680360850897 2023-04-01 14:54:13,306 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 266a372cef5b4350ace50fef50662e99, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680360850897 2023-04-01 14:54:13,306 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 3620e2be25b6424ba0cb1266bca5e984, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680360850943 2023-04-01 14:54:13,307 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cbbb03ffc372435f99d95e396a9eb0e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680360850943 2023-04-01 14:54:13,307 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 869835375fbb458c9c53e2768a0b206a, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680360853094 2023-04-01 14:54:13,307 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f927000734d14f1aa61656211fdb6023, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680360853113 2023-04-01 14:54:13,334 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2eee92d9536149526dfe0ebd08db5f#B#compaction#142 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:13,348 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:13,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:13,351 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 14:54:13,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:13,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:13,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:13,372 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:13,372 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:13,372 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:13,372 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:13,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,374 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:13,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:13,374 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,375 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,375 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:13,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:13,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:13,390 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2eee92d9536149526dfe0ebd08db5f#A#compaction#141 average throughput is 0.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:13,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:13,410 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:13,410 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:13,410 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:13,416 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/3805a3dbb3c24dcab5c8dcd3479b0a30 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/3805a3dbb3c24dcab5c8dcd3479b0a30 2023-04-01 14:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:13,416 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:13,421 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:13,421 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-01 14:54:13,421 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:13,421 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:13,422 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:13,422 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:13,422 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:13,422 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:13,432 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2eee92d9536149526dfe0ebd08db5f/B of 9a2eee92d9536149526dfe0ebd08db5f into 3805a3dbb3c24dcab5c8dcd3479b0a30(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:13,432 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:13,432 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., storeName=9a2eee92d9536149526dfe0ebd08db5f/B, priority=13, startTime=1680360853300; duration=0sec 2023-04-01 14:54:13,432 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:13,432 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:13,434 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:13,434 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 9a2eee92d9536149526dfe0ebd08db5f/C is initiating minor compaction (all files) 2023-04-01 14:54:13,434 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2eee92d9536149526dfe0ebd08db5f/C in TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:13,434 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/326c2c194b7d45ed8a32a2d522b15cea, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f44a229727614242b547b35565e2eafc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/02134c5d9a6a444f9072dd227ddaab50] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp, totalSize=34.8 K 2023-04-01 14:54:13,435 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 326c2c194b7d45ed8a32a2d522b15cea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680360850897 2023-04-01 14:54:13,435 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f44a229727614242b547b35565e2eafc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680360850943 2023-04-01 14:54:13,436 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 02134c5d9a6a444f9072dd227ddaab50, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680360853113 2023-04-01 14:54:13,456 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2eee92d9536149526dfe0ebd08db5f#C#compaction#143 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:13,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:13,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:13,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360913482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360913483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360913485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360913488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360913489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360913589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360913594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360913602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360913789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360913790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360913791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360913797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:13,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360913807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:13,871 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/59ec64728a2c44fe84cdf713da2f5f38 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/59ec64728a2c44fe84cdf713da2f5f38 2023-04-01 14:54:13,884 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2eee92d9536149526dfe0ebd08db5f/A of 9a2eee92d9536149526dfe0ebd08db5f into 59ec64728a2c44fe84cdf713da2f5f38(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:13,884 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:13,884 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., storeName=9a2eee92d9536149526dfe0ebd08db5f/A, priority=13, startTime=1680360853299; duration=0sec 2023-04-01 14:54:13,884 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:13,903 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/fb37408696d842b9a53ccfb97e36e50e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/fb37408696d842b9a53ccfb97e36e50e 2023-04-01 14:54:13,909 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401ea0fa4d3af2c40e793d0d1fc62bb543f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ea0fa4d3af2c40e793d0d1fc62bb543f 2023-04-01 14:54:13,912 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2eee92d9536149526dfe0ebd08db5f/C of 9a2eee92d9536149526dfe0ebd08db5f into fb37408696d842b9a53ccfb97e36e50e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:13,912 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:13,912 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., storeName=9a2eee92d9536149526dfe0ebd08db5f/C, priority=13, startTime=1680360853300; duration=0sec 2023-04-01 14:54:13,913 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:14,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:14,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360914097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:14,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360914098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:14,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360914113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:14,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360914293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:14,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360914300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,319 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=86, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/6d33cfd3a03d45c1a995ea4cd63e89c7 2023-04-01 14:54:14,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:14,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360914603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:14,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360914604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:14,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360914618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,752 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/9a8d21704c6a48a7bd9ef0d037d9ed0b 2023-04-01 14:54:14,792 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/9c213f1137ff444dbc863028c1d4d59b 2023-04-01 14:54:14,803 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/6d33cfd3a03d45c1a995ea4cd63e89c7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6d33cfd3a03d45c1a995ea4cd63e89c7 2023-04-01 14:54:14,810 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6d33cfd3a03d45c1a995ea4cd63e89c7, entries=150, sequenceid=86, filesize=25.1 K 2023-04-01 14:54:14,812 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/9a8d21704c6a48a7bd9ef0d037d9ed0b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/9a8d21704c6a48a7bd9ef0d037d9ed0b 2023-04-01 14:54:14,820 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/9a8d21704c6a48a7bd9ef0d037d9ed0b, entries=150, sequenceid=86, filesize=11.6 K 2023-04-01 14:54:14,822 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/9c213f1137ff444dbc863028c1d4d59b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/9c213f1137ff444dbc863028c1d4d59b 2023-04-01 14:54:14,827 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/9c213f1137ff444dbc863028c1d4d59b, entries=150, sequenceid=86, filesize=11.6 K 2023-04-01 14:54:14,828 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 9a2eee92d9536149526dfe0ebd08db5f in 1406ms, sequenceid=86, compaction requested=false 2023-04-01 14:54:14,828 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:14,828 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:14,828 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:14,828 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:14,828 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,828 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:14,828 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:14,830 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:14,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:14,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@851167f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:14,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:14,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:14,833 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:14,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:14,834 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:14,834 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:14,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:14,835 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:14,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:14,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:14,837 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:14,837 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:14,837 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:14,837 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:14,837 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:14,841 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:14,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:14,842 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,842 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:14,842 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:14,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:14,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:14,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:14,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:14,844 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:14,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:14,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:14,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:14,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:14,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:14,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:14,846 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:14,846 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:14,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:14,846 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:14,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:14,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:14,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:14,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:14,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:14,946 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:14,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:14,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:14,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:14,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:14,949 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:14,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:14,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:14,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:14,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:14,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:14,951 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:14,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:14,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:14,952 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:14,952 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:14,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:14,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:14,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:14,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:14,964 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:14,964 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:14,964 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 14:54:14,964 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:14,964 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:14,964 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:14,965 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:14,965 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:14,965 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:15,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:15,305 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:15,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:15,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360915329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:15,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:15,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360915330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:15,390 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304011cb1ececabe546f1ae805779d07155e1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011cb1ececabe546f1ae805779d07155e1 2023-04-01 14:54:15,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:15,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360915433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:15,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:15,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360915434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:15,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:15,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360915610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:15,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:15,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360915613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:15,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:15,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360915621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:15,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:15,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360915636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:15,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:15,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360915637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:15,797 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=101, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/0d4e9b459e11446e956e5b6a7f8730b4 2023-04-01 14:54:15,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:15,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360915939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:15,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:15,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360915939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,214 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/38dcac3fe4c64b51a7b7707a51d3aab7 2023-04-01 14:54:16,229 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/31d669309a314ded915c421c30096031 2023-04-01 14:54:16,233 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/0d4e9b459e11446e956e5b6a7f8730b4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/0d4e9b459e11446e956e5b6a7f8730b4 2023-04-01 14:54:16,239 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/0d4e9b459e11446e956e5b6a7f8730b4, entries=150, sequenceid=101, filesize=25.1 K 2023-04-01 14:54:16,247 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/38dcac3fe4c64b51a7b7707a51d3aab7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/38dcac3fe4c64b51a7b7707a51d3aab7 2023-04-01 14:54:16,263 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/38dcac3fe4c64b51a7b7707a51d3aab7, entries=150, sequenceid=101, filesize=11.6 K 2023-04-01 14:54:16,264 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/31d669309a314ded915c421c30096031 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/31d669309a314ded915c421c30096031 2023-04-01 14:54:16,278 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/31d669309a314ded915c421c30096031, entries=150, sequenceid=101, filesize=11.6 K 2023-04-01 14:54:16,280 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9a2eee92d9536149526dfe0ebd08db5f in 1316ms, sequenceid=101, compaction requested=true 2023-04-01 14:54:16,280 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:16,280 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:16,280 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:16,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:16,286 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:16,287 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:16,287 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:16,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:16,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@126382d1[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:16,291 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:16,291 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:16,292 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:16,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:16,292 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:16,292 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:16,292 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:16,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:16,294 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:16,299 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:16,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:16,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:16,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:16,299 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:16,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:16,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:16,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:16,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:16,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:16,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:16,304 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:16,308 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:16,308 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:16,308 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:16,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:16,308 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:16,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:16,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:16,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:16,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:16,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:16,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:16,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:16,310 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:16,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:16,310 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:16,410 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:16,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:16,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:16,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:16,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:16,414 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:16,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:16,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:16,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:16,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:16,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:16,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:16,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:16,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:16,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:16,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:16,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:16,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:16,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:16,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:16,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:16,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:16,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:16,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:16,418 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:16,418 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:16,418 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 14:54:16,418 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:16,418 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:16,418 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:16,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:16,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:16,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:16,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:16,444 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:16,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:16,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:16,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360916456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360916457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:16,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360916560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:16,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360916560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:16,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360916764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:16,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360916764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:16,841 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401859631f2e46a41238e34bb0ed6bc4593 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401859631f2e46a41238e34bb0ed6bc4593 2023-04-01 14:54:17,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:17,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360917067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:17,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360917067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,248 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=121, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/09d8a91723ac4340a6175fbbda7285d2 2023-04-01 14:54:17,259 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/711da8eb02a64d76aab06da0a73bc1b6 2023-04-01 14:54:17,271 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/e1a9216215834207b4f5eb0b45672f0a 2023-04-01 14:54:17,276 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/09d8a91723ac4340a6175fbbda7285d2 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/09d8a91723ac4340a6175fbbda7285d2 2023-04-01 14:54:17,280 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/09d8a91723ac4340a6175fbbda7285d2, entries=150, sequenceid=121, filesize=25.1 K 2023-04-01 14:54:17,282 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/711da8eb02a64d76aab06da0a73bc1b6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/711da8eb02a64d76aab06da0a73bc1b6 2023-04-01 14:54:17,287 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/711da8eb02a64d76aab06da0a73bc1b6, entries=150, sequenceid=121, filesize=11.6 K 2023-04-01 14:54:17,288 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/e1a9216215834207b4f5eb0b45672f0a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/e1a9216215834207b4f5eb0b45672f0a 2023-04-01 14:54:17,292 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/e1a9216215834207b4f5eb0b45672f0a, entries=150, sequenceid=121, filesize=11.6 K 2023-04-01 14:54:17,293 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9a2eee92d9536149526dfe0ebd08db5f in 875ms, sequenceid=121, compaction requested=true 2023-04-01 14:54:17,293 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:17,293 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:17,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:17,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:17,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:17,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:17,295 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:17,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:17,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:17,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18968b60[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:17,296 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:17,296 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:17,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:17,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:17,298 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:17,298 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:17,298 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:17,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:17,299 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:17,299 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,299 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:17,299 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:17,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:17,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:17,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:17,301 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:17,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:17,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:17,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:17,302 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:17,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:17,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:17,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:17,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:17,305 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:17,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:17,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:17,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:17,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:17,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:17,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:17,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:17,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:17,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:17,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:17,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:17,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:17,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:17,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:17,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:17,311 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:17,311 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:17,411 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:17,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:17,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:17,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:17,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:17,415 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:17,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:17,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:17,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:17,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:17,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:17,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:17,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:17,418 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,418 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:17,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:17,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:17,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:17,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:17,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:17,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:17,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:17,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:17,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:17,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:17,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:17,420 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 14:54:17,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:17,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:17,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:17,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:17,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:17,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:17,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:17,575 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:17,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:17,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360917596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:17,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360917597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:17,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360917619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,621 DEBUG [Thread-991] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:17,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:17,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360917623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,625 DEBUG [Thread-987] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:17,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:17,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360917629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,632 DEBUG [Thread-985] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:17,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:17,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360917699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:17,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360917700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,840 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401e69525ff3d66434a83067e84ccca0c4b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e69525ff3d66434a83067e84ccca0c4b 2023-04-01 14:54:17,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:17,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360917902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:17,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:17,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360917903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:18,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:18,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360918208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:18,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:18,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360918210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:18,247 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=133, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/284c1388517845bc94929d808a7537a8 2023-04-01 14:54:18,661 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/1484216803a749f28152105dc8475586 2023-04-01 14:54:18,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:18,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360918712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:18,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:18,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360918713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,072 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/6d6d55198c024bf69c2320b690849387 2023-04-01 14:54:19,077 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/284c1388517845bc94929d808a7537a8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/284c1388517845bc94929d808a7537a8 2023-04-01 14:54:19,081 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/284c1388517845bc94929d808a7537a8, entries=150, sequenceid=133, filesize=25.3 K 2023-04-01 14:54:19,082 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/1484216803a749f28152105dc8475586 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1484216803a749f28152105dc8475586 2023-04-01 14:54:19,085 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1484216803a749f28152105dc8475586, entries=150, sequenceid=133, filesize=11.8 K 2023-04-01 14:54:19,086 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/6d6d55198c024bf69c2320b690849387 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/6d6d55198c024bf69c2320b690849387 2023-04-01 14:54:19,092 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/6d6d55198c024bf69c2320b690849387, entries=150, sequenceid=133, filesize=11.8 K 2023-04-01 14:54:19,093 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9a2eee92d9536149526dfe0ebd08db5f in 1673ms, sequenceid=133, compaction requested=true 2023-04-01 14:54:19,093 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:19,093 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:19,093 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:19,093 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:19,093 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:19,095 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,095 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:19,095 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:19,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:19,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72bed754[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:19,097 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:19,097 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:19,098 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,098 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:19,099 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:19,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:19,099 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:19,099 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:19,099 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:19,101 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:19,101 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:19,101 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:19,101 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:19,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:19,103 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:19,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:19,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:19,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:19,103 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:19,110 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:19,110 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:19,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:19,110 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:19,111 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:19,112 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,114 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,114 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:19,114 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:19,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:19,114 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:19,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:19,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:19,115 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:19,114 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:19,114 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:19,115 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,115 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:19,115 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:19,115 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:19,115 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:19,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:19,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:19,215 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:19,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:19,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:19,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:19,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:19,219 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:19,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:19,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:19,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:19,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:19,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:19,220 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:19,220 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:19,221 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,221 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:19,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:19,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:19,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:19,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:19,222 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:19,222 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:19,222 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:19,222 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:19,222 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:19,222 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:19,222 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:19,222 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 14:54:19,222 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:19,223 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:19,223 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:19,223 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:19,223 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:19,223 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:19,236 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401b0b601e4117b41aaa84bbb785c3cb6c5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b0b601e4117b41aaa84bbb785c3cb6c5 2023-04-01 14:54:19,644 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=153, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/4cfa12a2947041f8b0e5de625ba76f7a 2023-04-01 14:54:19,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:19,716 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:19,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:19,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360919731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:19,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360919736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:19,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360919837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:19,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:19,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360919841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:20,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360920042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:20,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360920044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,056 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/cc4e9727772b432782e13c23e2018c00 2023-04-01 14:54:20,076 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/1ef9d789ea5e415186286908276e731f 2023-04-01 14:54:20,081 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/4cfa12a2947041f8b0e5de625ba76f7a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/4cfa12a2947041f8b0e5de625ba76f7a 2023-04-01 14:54:20,099 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/4cfa12a2947041f8b0e5de625ba76f7a, entries=150, sequenceid=153, filesize=25.3 K 2023-04-01 14:54:20,100 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/cc4e9727772b432782e13c23e2018c00 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cc4e9727772b432782e13c23e2018c00 2023-04-01 14:54:20,105 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cc4e9727772b432782e13c23e2018c00, entries=150, sequenceid=153, filesize=11.8 K 2023-04-01 14:54:20,106 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/1ef9d789ea5e415186286908276e731f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/1ef9d789ea5e415186286908276e731f 2023-04-01 14:54:20,112 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/1ef9d789ea5e415186286908276e731f, entries=150, sequenceid=153, filesize=11.8 K 2023-04-01 14:54:20,120 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9a2eee92d9536149526dfe0ebd08db5f in 898ms, sequenceid=153, compaction requested=true 2023-04-01 14:54:20,120 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:20,120 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:20,120 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:20,120 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:20,120 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,120 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:20,120 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:20,122 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:20,122 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:20,123 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,123 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725cd592[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:20,124 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:20,124 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,132 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:20,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:20,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:20,133 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:20,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:20,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:20,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:20,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:20,134 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:20,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:20,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:20,134 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:20,134 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:20,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:20,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:20,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:20,138 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:20,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:20,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,139 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:20,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,142 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:20,142 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:20,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:20,142 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:20,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:20,142 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:20,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:20,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:20,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:20,143 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:20,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:20,143 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:20,243 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:20,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:20,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:20,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:20,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:20,247 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:20,247 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:20,247 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:20,247 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:20,247 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,247 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:20,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,252 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:20,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:20,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,253 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:20,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:20,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:20,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:20,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:20,254 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:20,254 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:20,254 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:20,256 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:20,256 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:20,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:20,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:20,256 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 14:54:20,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:20,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:20,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:20,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:20,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:20,256 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:20,297 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040165eaf9c0d5d3480aa7eb58cbd20b986e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040165eaf9c0d5d3480aa7eb58cbd20b986e 2023-04-01 14:54:20,316 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/6b7dae358867407f94767316817401d5 2023-04-01 14:54:20,342 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/82b9020c310641ce9cbc5e0eb5c8e986 2023-04-01 14:54:20,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:20,359 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:20,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:20,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360920443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:20,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360920448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:20,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360920552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:20,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360920557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:20,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49020 deadline: 1680360920759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:20,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48976 deadline: 1680360920760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,830 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/36c323f0acba4bde8b5ce6570722e177 2023-04-01 14:54:20,835 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/6b7dae358867407f94767316817401d5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6b7dae358867407f94767316817401d5 2023-04-01 14:54:20,839 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6b7dae358867407f94767316817401d5, entries=150, sequenceid=165, filesize=25.3 K 2023-04-01 14:54:20,839 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/82b9020c310641ce9cbc5e0eb5c8e986 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/82b9020c310641ce9cbc5e0eb5c8e986 2023-04-01 14:54:20,843 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/82b9020c310641ce9cbc5e0eb5c8e986, entries=150, sequenceid=165, filesize=11.8 K 2023-04-01 14:54:20,844 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/36c323f0acba4bde8b5ce6570722e177 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/36c323f0acba4bde8b5ce6570722e177 2023-04-01 14:54:20,848 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/36c323f0acba4bde8b5ce6570722e177, entries=150, sequenceid=165, filesize=11.8 K 2023-04-01 14:54:20,848 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9a2eee92d9536149526dfe0ebd08db5f in 592ms, sequenceid=165, compaction requested=true 2023-04-01 14:54:20,848 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:20,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:20,852 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:20,852 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:20,852 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,852 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:20,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:20,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c27a402[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:20,853 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:20,853 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,855 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:20,855 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:20,855 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:20,855 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,855 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:20,856 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:20,856 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:20,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,856 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:20,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:20,858 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:20,858 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:20,858 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:20,858 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:20,858 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:20,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:20,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:20,860 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:20,861 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:20,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:20,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:20,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:20,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:20,867 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:20,868 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:20,868 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:20,968 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:20,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:20,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:20,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:20,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:20,971 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:20,971 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:20,971 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:20,971 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:20,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:20,973 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,973 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:20,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:20,973 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:20,973 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:20,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:20,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:20,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:20,975 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:20,975 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:20,975 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 14:54:20,975 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:20,975 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:20,975 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:20,975 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:20,976 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:20,976 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:20,988 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304016e5c47140d7d40d0b7f27a54483d59bc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304016e5c47140d7d40d0b7f27a54483d59bc 2023-04-01 14:54:20,993 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=185, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/494462fb51eb41c9b56098c16a56ecef 2023-04-01 14:54:21,003 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/84a22b4bc7e74a5491e1b8d5bf2db7d1 2023-04-01 14:54:21,013 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/3600c12b421f488583ce5aec272eda9d 2023-04-01 14:54:21,018 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/494462fb51eb41c9b56098c16a56ecef as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/494462fb51eb41c9b56098c16a56ecef 2023-04-01 14:54:21,022 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/494462fb51eb41c9b56098c16a56ecef, entries=150, sequenceid=185, filesize=25.3 K 2023-04-01 14:54:21,024 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/84a22b4bc7e74a5491e1b8d5bf2db7d1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/84a22b4bc7e74a5491e1b8d5bf2db7d1 2023-04-01 14:54:21,029 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/84a22b4bc7e74a5491e1b8d5bf2db7d1, entries=150, sequenceid=185, filesize=11.8 K 2023-04-01 14:54:21,030 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/3600c12b421f488583ce5aec272eda9d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/3600c12b421f488583ce5aec272eda9d 2023-04-01 14:54:21,035 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/3600c12b421f488583ce5aec272eda9d, entries=150, sequenceid=185, filesize=11.8 K 2023-04-01 14:54:21,036 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 9a2eee92d9536149526dfe0ebd08db5f in 61ms, sequenceid=185, compaction requested=true 2023-04-01 14:54:21,036 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:21,036 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:21,036 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:21,038 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,038 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,038 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,038 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5046ed6d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:21,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:21,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,044 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:21,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:21,045 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,045 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:21,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:21,046 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:21,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:21,049 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:21,049 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:21,049 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:21,049 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:21,049 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:21,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,051 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:21,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:21,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:21,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,057 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:21,057 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,057 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:21,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:21,136 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:54:21,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:21,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:21,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:21,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:21,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:21,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:21,157 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:21,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:21,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:21,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:21,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:21,161 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:21,161 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,161 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:21,161 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:21,161 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,161 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,163 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,163 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:21,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:21,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:21,164 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:21,164 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,164 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:21,165 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:21,165 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:21,165 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:21,166 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,166 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,167 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,167 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54c68446[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:21,169 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:21,169 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,171 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,171 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:21,171 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:21,172 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:21,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:21,173 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:21,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:21,175 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:21,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:21,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:21,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:21,175 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:21,177 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,177 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,177 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:21,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,183 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,183 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:21,182 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,183 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:21,183 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,183 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680360861186, max timestamp=1680360861210Num uniques 859; 2023-04-01 14:54:21,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-01 14:54:21,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680360861186, max timestamp=1680360861210Num uniques 859; 2023-04-01 14:54:21,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680360861186, max timestamp=1680360861210Num uniques 859; 2023-04-01 14:54:21,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-01 14:54:21,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-01 14:54:21,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:21,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360921238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:21,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360921241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,283 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:21,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:21,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:21,288 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:21,288 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:21,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:21,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,291 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:21,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:21,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:21,293 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,293 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:21,293 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:21,293 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:21,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:21,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:21,295 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:21,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a23b09[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:21,297 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:21,297 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:21,298 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:21,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:21,300 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,300 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:21,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:21,303 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:21,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:21,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:21,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:21,303 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:21,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:21,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:21,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,309 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:21,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,310 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,310 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:21,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:21,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360921342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:21,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360921347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,410 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:21,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:21,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:21,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:21,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:21,414 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:21,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:21,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:21,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,415 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:21,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:21,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:21,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:21,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:21,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:21,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:21,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:21,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:21,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:21,421 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,421 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,421 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,421 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e490ab[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:21,423 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:21,423 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,424 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:21,424 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:21,424 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:21,424 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,424 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:21,425 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,425 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:21,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,425 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,425 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:21,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:21,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:21,428 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:21,428 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:21,428 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:21,428 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:21,428 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:21,430 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,430 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,430 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,431 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,432 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,434 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:21,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:21,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,434 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:21,434 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:21,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,435 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,435 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,435 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,435 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,534 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:21,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:21,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:21,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:21,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:21,538 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:21,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:21,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:21,539 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,539 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,540 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,540 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,541 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:21,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:21,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:21,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:21,542 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:21,542 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,542 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:21,542 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:21,542 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:21,542 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,542 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:21,542 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:21,542 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,542 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:21,542 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:21,542 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,542 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:21,542 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:21,544 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,544 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,544 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,544 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@662e287f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:21,545 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:21,545 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,547 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,547 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:21,547 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:21,547 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,548 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:21,547 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:21,549 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:21,549 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,549 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,549 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:21,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360921547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:21,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360921552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:21,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:21,557 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:21,557 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:21,557 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:21,558 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:21,558 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:21,558 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040153aa691909be4c8f907c69b899773034 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040153aa691909be4c8f907c69b899773034 2023-04-01 14:54:21,559 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,559 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,559 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,560 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,562 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,569 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,569 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:21,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:21,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:21,569 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,569 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,569 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,570 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:21,570 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:21,569 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,570 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,570 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,570 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,570 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:21,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48994 deadline: 1680360921622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,628 DEBUG [Thread-991] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:21,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:21,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48986 deadline: 1680360921638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,643 DEBUG [Thread-987] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:21,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:21,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49034 deadline: 1680360921653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,659 DEBUG [Thread-985] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:21,670 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:21,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:21,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:21,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:21,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:21,673 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:21,673 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,673 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:21,673 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:21,674 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,674 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,676 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,676 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,676 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,676 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:21,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:21,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:21,678 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,678 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:21,678 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:21,678 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:21,680 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,680 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,680 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,680 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:21,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa7448[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:21,681 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:21,681 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,683 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:21,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:21,684 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:21,684 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,684 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,684 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:21,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:21,687 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:21,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:21,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:21,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:21,687 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:21,693 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,693 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,693 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,695 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,697 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,697 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,697 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,697 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,697 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,697 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:21,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:21,697 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:21,697 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,697 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:21,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,697 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:21,697 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,697 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,797 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:21,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:21,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:21,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:21,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:21,801 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:21,801 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,801 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:21,801 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:21,801 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,801 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,802 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,803 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,803 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,803 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:21,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:21,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:21,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:21,804 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:21,804 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,804 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:21,804 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:21,804 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:21,804 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,804 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:21,804 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:21,805 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:21,807 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57017de7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:21,809 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:21,809 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,814 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,814 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:21,814 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:21,814 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:21,814 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,814 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:21,815 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,815 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:21,815 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,815 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:21,817 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:21,817 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:21,817 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:21,817 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:21,818 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:21,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,820 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,823 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:21,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:21,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,823 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:21,824 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:21,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:21,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:21,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360921853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:21,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360921860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,924 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:21,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:21,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:21,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:21,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:21,928 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:21,928 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,928 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:21,929 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:21,929 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,929 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,932 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,932 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,932 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,932 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:21,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:21,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:21,935 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:21,935 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:21,935 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:21,935 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:21,935 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:21,935 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,935 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:21,936 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:21,936 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:21,936 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:21,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,940 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,941 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,941 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:21,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72c5ce64[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:21,944 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:21,944 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,945 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:21,946 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,946 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:21,948 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,948 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:21,948 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,948 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:21,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:21,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:21,952 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:21,952 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:21,953 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:21,953 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:21,953 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:21,958 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,958 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:21,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,958 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:21,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,959 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,960 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,962 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,962 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,962 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:21,962 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:21,962 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:21,962 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:21,962 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:21,962 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,962 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:21,962 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:21,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:21,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:21,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:21,963 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:21,963 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:21,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:21,969 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/3e94debeb4b344239e5ac976083800c2 2023-04-01 14:54:22,063 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:22,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:22,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:22,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:22,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:22,067 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:22,067 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,067 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:22,067 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:22,067 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,067 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,069 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,069 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,069 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:22,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:22,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:22,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:22,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:22,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:22,072 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,072 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:22,072 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:22,072 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:22,074 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:22,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:22,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65bf7d93[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:22,075 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:22,076 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,077 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:22,078 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:22,078 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:22,078 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:22,079 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,079 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,079 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,079 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:22,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:22,082 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:22,082 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:22,082 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:22,082 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:22,082 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:22,084 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,084 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,084 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,085 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,086 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:22,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:22,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:22,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,089 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,089 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:22,089 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:22,190 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:22,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:22,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:22,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:22,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:22,193 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:22,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:22,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:22,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,196 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,196 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,196 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,196 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:22,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:22,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:22,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:22,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:22,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:22,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:22,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:22,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:22,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:22,199 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,200 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,200 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:22,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30e24d58[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:22,201 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:22,201 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,202 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:22,202 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:22,203 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:22,203 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,203 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:22,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,204 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:22,204 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,204 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:22,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:22,206 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:22,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:22,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:22,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:22,206 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:22,211 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,211 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,211 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,212 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,213 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,215 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,215 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,215 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,215 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,215 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,216 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:22,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:22,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,216 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:22,216 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:22,316 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:22,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:22,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:22,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:22,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:22,319 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:22,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:22,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:22,320 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,320 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,321 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,321 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,321 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,321 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:22,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:22,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:22,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:22,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:22,323 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,323 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:22,323 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:22,323 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:22,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,325 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,325 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:22,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cd0e0e6[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:22,326 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:22,326 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,328 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:22,328 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,329 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:22,330 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:22,330 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,330 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,330 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:22,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:22,332 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:22,332 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:22,332 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:22,332 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:22,332 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:22,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,336 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,339 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,341 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,341 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,342 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,342 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,342 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,342 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,342 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:22,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:22,342 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,342 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,342 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,342 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,342 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:22,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:22,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:22,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360922357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,380 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/2ec96a8763944939a083d395c0a7a3e1 2023-04-01 14:54:22,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:22,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360922377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,442 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:22,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:22,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:22,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:22,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:22,446 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:22,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:22,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:22,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,448 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,448 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:22,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:22,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:22,450 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:22,450 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,450 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:22,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:22,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:22,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:22,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:22,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:22,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:22,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:22,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:22,452 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,452 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:22,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:22,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@303aa5f6[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:22,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:22,455 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:22,456 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,456 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:22,457 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:22,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:22,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:22,460 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:22,460 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:22,460 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:22,460 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:22,460 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:22,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:22,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:22,467 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:22,467 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:22,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:22,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,567 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:22,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:22,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:22,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:22,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:22,571 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:22,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:22,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:22,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,573 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,573 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,574 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,574 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:22,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:22,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:22,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:22,575 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,575 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:22,575 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:22,575 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:22,577 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:22,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ef70587[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:22,579 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:22,579 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,581 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:22,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:22,581 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,581 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:22,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:22,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:22,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:22,585 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:22,585 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:22,585 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:22,585 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:22,585 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:22,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,588 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,594 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:22,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:22,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:22,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,594 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,594 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:22,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,695 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:22,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:22,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:22,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:22,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:22,698 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:22,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:22,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:22,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,700 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,700 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,700 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,700 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:22,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:22,701 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:22,702 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,702 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:22,702 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:22,702 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:22,704 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c5d058c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:22,705 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:22,705 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,706 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,707 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:22,707 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:22,707 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:22,707 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,707 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:22,709 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,709 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:22,709 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,710 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:22,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:22,712 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:22,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:22,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:22,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:22,712 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:22,714 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,714 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,714 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:22,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,714 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:22,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,717 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,717 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,717 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,717 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,717 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:22,717 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,717 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:22,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:22,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:22,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:22,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:22,718 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:22,718 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:22,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:22,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/17fe10ed4272433b83c58e09fd0b4bb9 2023-04-01 14:54:22,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/3e94debeb4b344239e5ac976083800c2 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3e94debeb4b344239e5ac976083800c2 2023-04-01 14:54:22,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3e94debeb4b344239e5ac976083800c2, entries=350, sequenceid=195, filesize=52.6 K 2023-04-01 14:54:22,801 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/2ec96a8763944939a083d395c0a7a3e1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/2ec96a8763944939a083d395c0a7a3e1 2023-04-01 14:54:22,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/2ec96a8763944939a083d395c0a7a3e1, entries=150, sequenceid=195, filesize=11.8 K 2023-04-01 14:54:22,805 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/17fe10ed4272433b83c58e09fd0b4bb9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/17fe10ed4272433b83c58e09fd0b4bb9 2023-04-01 14:54:22,809 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/17fe10ed4272433b83c58e09fd0b4bb9, entries=150, sequenceid=195, filesize=11.8 K 2023-04-01 14:54:22,810 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 9a2eee92d9536149526dfe0ebd08db5f in 1674ms, sequenceid=195, compaction requested=true 2023-04-01 14:54:22,810 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:22,810 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:22,810 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-01 14:54:22,810 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-01 14:54:22,810 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:22,810 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:22,817 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 260531 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-01 14:54:22,817 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 9a2eee92d9536149526dfe0ebd08db5f/A is initiating minor compaction (all files) 2023-04-01 14:54:22,817 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2eee92d9536149526dfe0ebd08db5f/A in TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,817 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/59ec64728a2c44fe84cdf713da2f5f38, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6d33cfd3a03d45c1a995ea4cd63e89c7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/0d4e9b459e11446e956e5b6a7f8730b4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/09d8a91723ac4340a6175fbbda7285d2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/284c1388517845bc94929d808a7537a8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/4cfa12a2947041f8b0e5de625ba76f7a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6b7dae358867407f94767316817401d5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/494462fb51eb41c9b56098c16a56ecef, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3e94debeb4b344239e5ac976083800c2] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp, totalSize=254.4 K 2023-04-01 14:54:22,818 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:22,818 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 59ec64728a2c44fe84cdf713da2f5f38, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680360853113 2023-04-01 14:54:22,818 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107827 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-01 14:54:22,819 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 9a2eee92d9536149526dfe0ebd08db5f/B is initiating minor compaction (all files) 2023-04-01 14:54:22,819 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2eee92d9536149526dfe0ebd08db5f/B in TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,819 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 6d33cfd3a03d45c1a995ea4cd63e89c7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680360853166 2023-04-01 14:54:22,819 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/3805a3dbb3c24dcab5c8dcd3479b0a30, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/9a8d21704c6a48a7bd9ef0d037d9ed0b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/38dcac3fe4c64b51a7b7707a51d3aab7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/711da8eb02a64d76aab06da0a73bc1b6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1484216803a749f28152105dc8475586, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cc4e9727772b432782e13c23e2018c00, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/82b9020c310641ce9cbc5e0eb5c8e986, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/84a22b4bc7e74a5491e1b8d5bf2db7d1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/2ec96a8763944939a083d395c0a7a3e1] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp, totalSize=105.3 K 2023-04-01 14:54:22,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:22,819 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 0d4e9b459e11446e956e5b6a7f8730b4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680360853465 2023-04-01 14:54:22,820 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3805a3dbb3c24dcab5c8dcd3479b0a30, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680360853113 2023-04-01 14:54:22,820 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 09d8a91723ac4340a6175fbbda7285d2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680360855325 2023-04-01 14:54:22,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:22,822 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9a8d21704c6a48a7bd9ef0d037d9ed0b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680360853166 2023-04-01 14:54:22,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:22,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:22,822 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:22,822 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 284c1388517845bc94929d808a7537a8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680360856452 2023-04-01 14:54:22,823 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 38dcac3fe4c64b51a7b7707a51d3aab7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680360853465 2023-04-01 14:54:22,822 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,823 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:22,823 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:22,823 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 711da8eb02a64d76aab06da0a73bc1b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680360855325 2023-04-01 14:54:22,823 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 4cfa12a2947041f8b0e5de625ba76f7a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680360857584 2023-04-01 14:54:22,823 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,823 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,824 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1484216803a749f28152105dc8475586, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680360856452 2023-04-01 14:54:22,824 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 6b7dae358867407f94767316817401d5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680360859719 2023-04-01 14:54:22,824 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cc4e9727772b432782e13c23e2018c00, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680360857584 2023-04-01 14:54:22,825 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 494462fb51eb41c9b56098c16a56ecef, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680360860432 2023-04-01 14:54:22,825 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:22,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:22,825 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:22,825 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 82b9020c310641ce9cbc5e0eb5c8e986, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680360859719 2023-04-01 14:54:22,825 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 3e94debeb4b344239e5ac976083800c2, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680360861090 2023-04-01 14:54:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:22,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:22,826 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 84a22b4bc7e74a5491e1b8d5bf2db7d1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680360860432 2023-04-01 14:54:22,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:22,826 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2ec96a8763944939a083d395c0a7a3e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680360861097 2023-04-01 14:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:22,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:22,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:22,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:22,839 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:22,839 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:22,839 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:22,839 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:22,839 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:22,839 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:22,839 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:22,839 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-01 14:54:22,839 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:22,840 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:22,840 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:22,840 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:22,840 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:22,840 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:22,846 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2eee92d9536149526dfe0ebd08db5f#B#compaction#169 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:22,856 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2eee92d9536149526dfe0ebd08db5f#A#compaction#168 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:23,260 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/380f62aab1074e0aaeb2105266f36a9e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/380f62aab1074e0aaeb2105266f36a9e 2023-04-01 14:54:23,264 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 9a2eee92d9536149526dfe0ebd08db5f/B of 9a2eee92d9536149526dfe0ebd08db5f into 380f62aab1074e0aaeb2105266f36a9e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:23,264 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:23,264 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., storeName=9a2eee92d9536149526dfe0ebd08db5f/B, priority=7, startTime=1680360862810; duration=0sec 2023-04-01 14:54:23,264 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:23,264 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-01 14:54:23,268 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107827 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-01 14:54:23,268 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 9a2eee92d9536149526dfe0ebd08db5f/C is initiating minor compaction (all files) 2023-04-01 14:54:23,268 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2eee92d9536149526dfe0ebd08db5f/C in TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:23,268 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/fb37408696d842b9a53ccfb97e36e50e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/9c213f1137ff444dbc863028c1d4d59b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/31d669309a314ded915c421c30096031, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/e1a9216215834207b4f5eb0b45672f0a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/6d6d55198c024bf69c2320b690849387, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/1ef9d789ea5e415186286908276e731f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/36c323f0acba4bde8b5ce6570722e177, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/3600c12b421f488583ce5aec272eda9d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/17fe10ed4272433b83c58e09fd0b4bb9] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp, totalSize=105.3 K 2023-04-01 14:54:23,269 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting fb37408696d842b9a53ccfb97e36e50e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680360853113 2023-04-01 14:54:23,270 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9c213f1137ff444dbc863028c1d4d59b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680360853166 2023-04-01 14:54:23,271 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 31d669309a314ded915c421c30096031, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680360853465 2023-04-01 14:54:23,271 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e1a9216215834207b4f5eb0b45672f0a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680360855325 2023-04-01 14:54:23,272 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6d6d55198c024bf69c2320b690849387, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680360856452 2023-04-01 14:54:23,272 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1ef9d789ea5e415186286908276e731f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680360857584 2023-04-01 14:54:23,273 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 36c323f0acba4bde8b5ce6570722e177, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680360859719 2023-04-01 14:54:23,273 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3600c12b421f488583ce5aec272eda9d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680360860432 2023-04-01 14:54:23,273 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304018d41936308c04235a2f9aede0b51dbd9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018d41936308c04235a2f9aede0b51dbd9 2023-04-01 14:54:23,274 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 17fe10ed4272433b83c58e09fd0b4bb9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680360861097 2023-04-01 14:54:23,290 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=230, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/d08a532a046447a1b7e5176c2ba5bfe5 2023-04-01 14:54:23,295 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2eee92d9536149526dfe0ebd08db5f#C#compaction#171 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:23,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:23,372 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:23,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:23,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360923396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:23,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:23,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360923396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:23,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:23,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360923500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:23,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:23,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360923500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:23,684 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/517afc3d40d0421d8e326a2b3bc3e552 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/517afc3d40d0421d8e326a2b3bc3e552 2023-04-01 14:54:23,689 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 9a2eee92d9536149526dfe0ebd08db5f/A of 9a2eee92d9536149526dfe0ebd08db5f into 517afc3d40d0421d8e326a2b3bc3e552(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:23,689 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:23,690 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., storeName=9a2eee92d9536149526dfe0ebd08db5f/A, priority=7, startTime=1680360862810; duration=0sec 2023-04-01 14:54:23,690 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:23,702 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/1fcc4686908549dcae27de30678deaeb 2023-04-01 14:54:23,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:23,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360923705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:23,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:23,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360923709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:23,717 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/3c42b5e9fd0d47c58d088ddaf9953915 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/3c42b5e9fd0d47c58d088ddaf9953915 2023-04-01 14:54:23,723 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 9a2eee92d9536149526dfe0ebd08db5f/C of 9a2eee92d9536149526dfe0ebd08db5f into 3c42b5e9fd0d47c58d088ddaf9953915(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:23,723 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:23,723 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., storeName=9a2eee92d9536149526dfe0ebd08db5f/C, priority=7, startTime=1680360862810; duration=0sec 2023-04-01 14:54:23,723 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:24,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:24,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360924012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:24,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360924012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,134 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/f47d8c5c9ff54232a59cb68ce8b8e28b 2023-04-01 14:54:24,139 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/d08a532a046447a1b7e5176c2ba5bfe5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/d08a532a046447a1b7e5176c2ba5bfe5 2023-04-01 14:54:24,144 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/d08a532a046447a1b7e5176c2ba5bfe5, entries=150, sequenceid=230, filesize=25.3 K 2023-04-01 14:54:24,145 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/1fcc4686908549dcae27de30678deaeb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1fcc4686908549dcae27de30678deaeb 2023-04-01 14:54:24,160 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1fcc4686908549dcae27de30678deaeb, entries=150, sequenceid=230, filesize=11.8 K 2023-04-01 14:54:24,161 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/f47d8c5c9ff54232a59cb68ce8b8e28b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f47d8c5c9ff54232a59cb68ce8b8e28b 2023-04-01 14:54:24,172 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f47d8c5c9ff54232a59cb68ce8b8e28b, entries=150, sequenceid=230, filesize=11.8 K 2023-04-01 14:54:24,173 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=53.67 KB/54960 for 9a2eee92d9536149526dfe0ebd08db5f in 1334ms, sequenceid=230, compaction requested=false 2023-04-01 14:54:24,173 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:24,173 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:24,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:24,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:24,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:24,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:24,175 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:24,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:24,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:24,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:24,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:24,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:24,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67e10331[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:24,179 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:24,179 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:24,180 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:24,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:24,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:24,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:24,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:24,180 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,180 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:24,183 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:24,183 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:24,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:24,184 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:24,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:24,187 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:24,187 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:24,188 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:24,188 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:24,188 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:24,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:24,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:24,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:24,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:24,190 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:24,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:24,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:24,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:24,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:24,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:24,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:24,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:24,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:24,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:24,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:24,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:24,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:24,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:24,196 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:24,196 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:24,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:24,296 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:24,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:24,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:24,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:24,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:24,300 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:24,300 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:24,300 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:24,300 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:24,301 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:24,301 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:24,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:24,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:24,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:24,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:24,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:24,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:24,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:24,304 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:24,304 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:24,304 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-01 14:54:24,304 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:24,304 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:24,304 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:24,304 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:24,304 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:24,304 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:24,316 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401ebd46c26e5854e5996bf87db01941e0a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ebd46c26e5854e5996bf87db01941e0a 2023-04-01 14:54:24,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:24,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:24,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680360864536, max timestamp=1680360864547Num uniques 859; 2023-04-01 14:54:24,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:54:24,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680360864536, max timestamp=1680360864547Num uniques 859; 2023-04-01 14:54:24,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:54:24,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680360864536, max timestamp=1680360864547Num uniques 859; 2023-04-01 14:54:24,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:54:24,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:24,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360924583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:24,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360924584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:24,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360924688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:24,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360924689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,722 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=244, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/c1f9d12c218b4564aba3a1efb50a0285 2023-04-01 14:54:24,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:24,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360924890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:24,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:24,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360924892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,137 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/a84f2a332032465fa3f9d51f08217a98 2023-04-01 14:54:25,149 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/0fc9d1a87b084e118ca502a25dc1cc0d 2023-04-01 14:54:25,154 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/c1f9d12c218b4564aba3a1efb50a0285 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c1f9d12c218b4564aba3a1efb50a0285 2023-04-01 14:54:25,157 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c1f9d12c218b4564aba3a1efb50a0285, entries=150, sequenceid=244, filesize=25.3 K 2023-04-01 14:54:25,158 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/a84f2a332032465fa3f9d51f08217a98 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/a84f2a332032465fa3f9d51f08217a98 2023-04-01 14:54:25,162 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/a84f2a332032465fa3f9d51f08217a98, entries=150, sequenceid=244, filesize=11.8 K 2023-04-01 14:54:25,163 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/0fc9d1a87b084e118ca502a25dc1cc0d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/0fc9d1a87b084e118ca502a25dc1cc0d 2023-04-01 14:54:25,167 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/0fc9d1a87b084e118ca502a25dc1cc0d, entries=150, sequenceid=244, filesize=11.8 K 2023-04-01 14:54:25,167 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9a2eee92d9536149526dfe0ebd08db5f in 863ms, sequenceid=244, compaction requested=true 2023-04-01 14:54:25,168 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:25,168 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:25,168 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:25,168 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:25,168 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,168 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:25,168 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:25,169 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,169 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:25,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:25,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2910f2e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:25,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:25,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,172 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:25,172 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:25,172 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:25,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:25,174 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:25,174 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:25,174 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,174 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:25,176 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:25,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:25,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:25,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:25,177 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:25,178 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,178 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,178 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,180 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:25,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:25,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:25,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:25,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:25,185 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:25,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:25,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,186 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:25,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:25,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,186 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:25,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:25,195 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-01 14:54:25,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:25,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:25,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:25,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:25,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:25,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:25,208 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401e16d16988862464a8558c21af21bd2bf to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e16d16988862464a8558c21af21bd2bf 2023-04-01 14:54:25,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:25,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360925206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:25,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360925207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,218 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=280, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/c34fd9d97f144fcbadfc0d3bfdd4d76a 2023-04-01 14:54:25,286 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:25,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:25,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:25,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:25,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:25,289 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:25,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:25,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:25,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:25,290 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,290 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,292 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:25,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:25,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,293 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:25,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:25,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:25,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:25,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:25,294 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:25,294 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:25,294 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:25,294 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:25,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:25,296 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,296 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,296 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,296 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:25,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14dce9c6[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:25,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:25,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,299 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,299 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:25,299 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:25,299 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:25,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:25,301 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:25,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:25,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:25,303 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:25,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:25,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:25,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:25,304 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:25,306 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,306 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,306 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:25,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:25,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:25,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:25,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:25,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:25,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:25,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360925310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,312 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:25,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:25,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360925312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:25,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:25,317 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:25,317 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:25,417 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:25,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:25,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:25,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:25,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:25,420 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:25,420 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:25,420 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:25,420 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:25,420 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,420 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,421 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,421 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:25,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:25,422 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,422 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:25,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:25,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:25,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:25,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:25,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:25,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:25,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:25,423 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:25,423 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:25,423 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:25,423 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:25,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:25,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:25,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:25,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:25,425 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,425 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,426 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,426 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:25,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:25,427 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:25,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7049efb4[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:25,427 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,433 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:25,433 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,433 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:25,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,434 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:25,435 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,435 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:25,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:25,440 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:25,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:25,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:25,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:25,440 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:25,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:25,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,443 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,444 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,446 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,446 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:25,446 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,446 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:25,446 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:25,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:25,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:25,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:25,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:25,447 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:25,447 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:25,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:25,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:25,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360925514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:25,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360925514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,547 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:25,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:25,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:25,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:25,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:25,551 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:25,551 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:25,551 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:25,551 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:25,551 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,551 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,553 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:25,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:25,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:25,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:25,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:25,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:25,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:25,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:25,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:25,556 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:25,556 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:25,556 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:25,556 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:25,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:25,557 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,558 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,559 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,559 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:25,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fde9207[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:25,560 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:25,560 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,561 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:25,561 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:25,561 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,561 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:25,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,563 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:25,563 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,563 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:25,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:25,565 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:25,565 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:25,565 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:25,565 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:25,565 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:25,568 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,568 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,568 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:25,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:25,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:25,568 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,569 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:25,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:25,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:25,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:25,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:25,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:25,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,572 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,572 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,572 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,572 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:25,572 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,572 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:25,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:25,572 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:25,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:25,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/1d417e3946474e03821e8d32d0c38ed2 2023-04-01 14:54:25,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/d404c9aa8a224651a52bc2aa57a51208 2023-04-01 14:54:25,648 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/c34fd9d97f144fcbadfc0d3bfdd4d76a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c34fd9d97f144fcbadfc0d3bfdd4d76a 2023-04-01 14:54:25,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c34fd9d97f144fcbadfc0d3bfdd4d76a, entries=150, sequenceid=280, filesize=25.4 K 2023-04-01 14:54:25,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/1d417e3946474e03821e8d32d0c38ed2 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1d417e3946474e03821e8d32d0c38ed2 2023-04-01 14:54:25,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1d417e3946474e03821e8d32d0c38ed2, entries=150, sequenceid=280, filesize=11.9 K 2023-04-01 14:54:25,658 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/d404c9aa8a224651a52bc2aa57a51208 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/d404c9aa8a224651a52bc2aa57a51208 2023-04-01 14:54:25,662 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/d404c9aa8a224651a52bc2aa57a51208, entries=150, sequenceid=280, filesize=11.9 K 2023-04-01 14:54:25,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 9a2eee92d9536149526dfe0ebd08db5f in 467ms, sequenceid=280, compaction requested=true 2023-04-01 14:54:25,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:25,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:25,663 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:54:25,663 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:54:25,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:25,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:25,664 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48704 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:54:25,665 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104144 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:54:25,665 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 9a2eee92d9536149526dfe0ebd08db5f/B is initiating minor compaction (all files) 2023-04-01 14:54:25,665 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 9a2eee92d9536149526dfe0ebd08db5f/A is initiating minor compaction (all files) 2023-04-01 14:54:25,665 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2eee92d9536149526dfe0ebd08db5f/B in TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:25,665 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2eee92d9536149526dfe0ebd08db5f/A in TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:25,665 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/380f62aab1074e0aaeb2105266f36a9e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1fcc4686908549dcae27de30678deaeb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/a84f2a332032465fa3f9d51f08217a98, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1d417e3946474e03821e8d32d0c38ed2] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp, totalSize=47.6 K 2023-04-01 14:54:25,665 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/517afc3d40d0421d8e326a2b3bc3e552, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/d08a532a046447a1b7e5176c2ba5bfe5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c1f9d12c218b4564aba3a1efb50a0285, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c34fd9d97f144fcbadfc0d3bfdd4d76a] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp, totalSize=101.7 K 2023-04-01 14:54:25,665 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 380f62aab1074e0aaeb2105266f36a9e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680360861097 2023-04-01 14:54:25,666 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 517afc3d40d0421d8e326a2b3bc3e552, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680360861097 2023-04-01 14:54:25,666 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1fcc4686908549dcae27de30678deaeb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680360861230 2023-04-01 14:54:25,666 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d08a532a046447a1b7e5176c2ba5bfe5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680360861230 2023-04-01 14:54:25,666 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a84f2a332032465fa3f9d51f08217a98, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680360863386 2023-04-01 14:54:25,667 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting c1f9d12c218b4564aba3a1efb50a0285, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680360863386 2023-04-01 14:54:25,668 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1d417e3946474e03821e8d32d0c38ed2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680360864576 2023-04-01 14:54:25,669 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting c34fd9d97f144fcbadfc0d3bfdd4d76a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680360864576 2023-04-01 14:54:25,672 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:25,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:25,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:25,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:25,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:25,677 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:25,677 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:25,677 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:25,677 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:25,678 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,678 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,679 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:25,679 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,680 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,680 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:25,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:25,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:25,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:25,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:25,686 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:25,686 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:25,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:25,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:25,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:25,687 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:25,687 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:25,687 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:54:25,688 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:25,688 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:25,688 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:25,688 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:25,688 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:25,688 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:25,692 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2eee92d9536149526dfe0ebd08db5f#B#compaction#180 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:25,735 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2eee92d9536149526dfe0ebd08db5f#A#compaction#181 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:25,760 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/c5be4369e8f04f058a4888185d2936d0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/c5be4369e8f04f058a4888185d2936d0 2023-04-01 14:54:25,767 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304017911ad2c2a714f508ee4e8fbc51a4a64 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017911ad2c2a714f508ee4e8fbc51a4a64 2023-04-01 14:54:25,767 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2eee92d9536149526dfe0ebd08db5f/B of 9a2eee92d9536149526dfe0ebd08db5f into c5be4369e8f04f058a4888185d2936d0(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:25,768 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:25,768 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., storeName=9a2eee92d9536149526dfe0ebd08db5f/B, priority=12, startTime=1680360865663; duration=0sec 2023-04-01 14:54:25,768 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:25,768 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:54:25,773 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48704 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:54:25,773 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 9a2eee92d9536149526dfe0ebd08db5f/C is initiating minor compaction (all files) 2023-04-01 14:54:25,773 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2eee92d9536149526dfe0ebd08db5f/C in TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:25,773 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/3c42b5e9fd0d47c58d088ddaf9953915, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f47d8c5c9ff54232a59cb68ce8b8e28b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/0fc9d1a87b084e118ca502a25dc1cc0d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/d404c9aa8a224651a52bc2aa57a51208] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp, totalSize=47.6 K 2023-04-01 14:54:25,774 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3c42b5e9fd0d47c58d088ddaf9953915, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680360861097 2023-04-01 14:54:25,774 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f47d8c5c9ff54232a59cb68ce8b8e28b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680360861230 2023-04-01 14:54:25,775 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 0fc9d1a87b084e118ca502a25dc1cc0d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1680360863386 2023-04-01 14:54:25,775 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting d404c9aa8a224651a52bc2aa57a51208, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680360864576 2023-04-01 14:54:25,791 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2eee92d9536149526dfe0ebd08db5f#C#compaction#183 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:25,799 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/0abc8fda0d23467581d295986e34ed94 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/0abc8fda0d23467581d295986e34ed94 2023-04-01 14:54:25,803 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2eee92d9536149526dfe0ebd08db5f/C of 9a2eee92d9536149526dfe0ebd08db5f into 0abc8fda0d23467581d295986e34ed94(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:25,803 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:25,803 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., storeName=9a2eee92d9536149526dfe0ebd08db5f/C, priority=12, startTime=1680360865663; duration=0sec 2023-04-01 14:54:25,803 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:25,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:25,819 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:25,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360865831, max timestamp=1680360865841Num uniques 859; 2023-04-01 14:54:25,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:54:25,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360865831, max timestamp=1680360865841Num uniques 859; 2023-04-01 14:54:25,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680360865831, max timestamp=1680360865841Num uniques 859; 2023-04-01 14:54:25,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:54:25,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-01 14:54:25,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:25,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360925870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:25,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360925871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:25,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360925972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:25,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:25,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360925973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:26,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:26,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360926175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:26,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:26,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360926178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:26,185 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/d2750d79f13548f8a91aa590097c9091 2023-04-01 14:54:26,189 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/062fd4cae1444299a5ac8f7a246c9a45 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/062fd4cae1444299a5ac8f7a246c9a45 2023-04-01 14:54:26,198 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2eee92d9536149526dfe0ebd08db5f/A of 9a2eee92d9536149526dfe0ebd08db5f into 062fd4cae1444299a5ac8f7a246c9a45(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:26,198 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:26,198 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f., storeName=9a2eee92d9536149526dfe0ebd08db5f/A, priority=12, startTime=1680360865663; duration=0sec 2023-04-01 14:54:26,198 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360926477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360926479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:26,598 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/b2169ecdf15e4e16867abb13497dab85 2023-04-01 14:54:26,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:26,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360926981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:26,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:26,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360926984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,012 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/4bdcc450ef3a4ca7b6853f089de4624e 2023-04-01 14:54:27,017 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/d2750d79f13548f8a91aa590097c9091 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/d2750d79f13548f8a91aa590097c9091 2023-04-01 14:54:27,021 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/d2750d79f13548f8a91aa590097c9091, entries=150, sequenceid=290, filesize=25.4 K 2023-04-01 14:54:27,022 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/b2169ecdf15e4e16867abb13497dab85 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/b2169ecdf15e4e16867abb13497dab85 2023-04-01 14:54:27,026 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/b2169ecdf15e4e16867abb13497dab85, entries=150, sequenceid=290, filesize=11.9 K 2023-04-01 14:54:27,027 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/4bdcc450ef3a4ca7b6853f089de4624e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/4bdcc450ef3a4ca7b6853f089de4624e 2023-04-01 14:54:27,040 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/4bdcc450ef3a4ca7b6853f089de4624e, entries=150, sequenceid=290, filesize=11.9 K 2023-04-01 14:54:27,041 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9a2eee92d9536149526dfe0ebd08db5f in 1354ms, sequenceid=290, compaction requested=false 2023-04-01 14:54:27,041 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:27,041 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:27,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:27,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:27,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:27,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:27,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15bc06ad[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:27,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:27,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:27,055 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:27,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:27,055 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:27,055 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:27,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:27,055 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:27,057 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:27,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:27,060 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:27,060 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:27,060 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:27,060 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:27,060 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:27,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:27,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:27,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:27,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:27,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:27,063 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:27,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:27,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:27,069 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:27,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:27,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:27,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:27,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:27,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:27,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:27,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:27,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:27,069 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:27,069 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:27,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:27,169 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:27,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:27,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:27,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:27,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:27,173 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:27,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:27,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:27,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:27,174 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:27,174 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:27,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,175 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:27,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:27,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:27,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:27,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:27,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:27,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:27,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:27,177 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:27,177 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:27,177 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-01 14:54:27,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:27,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:27,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:27,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:27,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:27,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:27,599 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304013ae319efece940e6bc2d2cc56635907a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013ae319efece940e6bc2d2cc56635907a 2023-04-01 14:54:27,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:27,989 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:28,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:28,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360927999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:28,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360928000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,016 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=328, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/8dc9502050be4120ab6633f840133692 2023-04-01 14:54:28,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:28,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360928101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:28,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360928101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:28,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360928305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:28,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360928306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,437 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/bee07b3df9f84e42a06501863d56cbc0 2023-04-01 14:54:28,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:28,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360928607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:28,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360928608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,849 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/84cb9f16078541149f094d7c7751ba9b 2023-04-01 14:54:28,855 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/8dc9502050be4120ab6633f840133692 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/8dc9502050be4120ab6633f840133692 2023-04-01 14:54:28,861 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/8dc9502050be4120ab6633f840133692, entries=150, sequenceid=328, filesize=25.4 K 2023-04-01 14:54:28,862 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/bee07b3df9f84e42a06501863d56cbc0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/bee07b3df9f84e42a06501863d56cbc0 2023-04-01 14:54:28,868 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/bee07b3df9f84e42a06501863d56cbc0, entries=150, sequenceid=328, filesize=11.9 K 2023-04-01 14:54:28,872 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/84cb9f16078541149f094d7c7751ba9b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/84cb9f16078541149f094d7c7751ba9b 2023-04-01 14:54:28,876 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/84cb9f16078541149f094d7c7751ba9b, entries=150, sequenceid=328, filesize=11.9 K 2023-04-01 14:54:28,877 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 9a2eee92d9536149526dfe0ebd08db5f in 1700ms, sequenceid=328, compaction requested=true 2023-04-01 14:54:28,877 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:28,877 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:28,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:28,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:28,896 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:28,897 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:28,897 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:28,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:28,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@665d5f05[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:28,908 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:28,908 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:28,909 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:28,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:28,910 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,910 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:28,910 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:28,910 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:28,910 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:28,911 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:28,911 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:28,911 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,911 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:28,914 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:28,914 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:28,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:28,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:28,915 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:28,917 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:28,917 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:28,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:28,917 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:28,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:28,918 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:28,920 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,922 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,922 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:28,922 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:28,922 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:28,923 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:28,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:28,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:28,922 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:28,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:28,923 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:28,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:28,923 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:28,923 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:28,923 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:28,923 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:28,923 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:28,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:29,024 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:29,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:29,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:29,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:29,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:29,037 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:29,037 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:29,037 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:29,037 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:29,037 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:29,037 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:29,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:29,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:29,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:29,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:29,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:29,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:29,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:29,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:29,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:29,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:29,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:29,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:29,056 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:29,064 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:29,064 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:29,064 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:54:29,064 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:29,065 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:29,065 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:29,065 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:29,065 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:29,065 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:29,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:29,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. as already flushing 2023-04-01 14:54:29,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680360869151, max timestamp=1680360869168Num uniques 859; 2023-04-01 14:54:29,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680360869151, max timestamp=1680360869168Num uniques 859; 2023-04-01 14:54:29,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-01 14:54:29,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680360869151, max timestamp=1680360869168Num uniques 859; 2023-04-01 14:54:29,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-01 14:54:29,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-01 14:54:29,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:29,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360929219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:29,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:29,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360929220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:29,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:29,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360929324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:29,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:29,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360929324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:29,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:29,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360929526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:29,529 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401ce3f50ef2b174e508d62da291dc0dbb7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ce3f50ef2b174e508d62da291dc0dbb7 2023-04-01 14:54:29,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:29,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360929527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:29,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:29,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360929827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:29,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:29,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360929836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:29,940 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=338, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/7d4d62bc978148f3bf897211ca7a9d65 2023-04-01 14:54:29,982 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/12caac5a064d4efcb7ab35927d576638 2023-04-01 14:54:30,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:30,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48976 deadline: 1680360930336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:30,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49020 deadline: 1680360930345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,397 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/2a448917fcc94645a36f85bf2620ed49 2023-04-01 14:54:30,401 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/7d4d62bc978148f3bf897211ca7a9d65 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/7d4d62bc978148f3bf897211ca7a9d65 2023-04-01 14:54:30,405 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/7d4d62bc978148f3bf897211ca7a9d65, entries=100, sequenceid=338, filesize=18.6 K 2023-04-01 14:54:30,406 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/12caac5a064d4efcb7ab35927d576638 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/12caac5a064d4efcb7ab35927d576638 2023-04-01 14:54:30,411 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/12caac5a064d4efcb7ab35927d576638, entries=100, sequenceid=338, filesize=9.5 K 2023-04-01 14:54:30,412 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/2a448917fcc94645a36f85bf2620ed49 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/2a448917fcc94645a36f85bf2620ed49 2023-04-01 14:54:30,415 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/2a448917fcc94645a36f85bf2620ed49, entries=100, sequenceid=338, filesize=9.5 K 2023-04-01 14:54:30,416 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 9a2eee92d9536149526dfe0ebd08db5f in 1352ms, sequenceid=338, compaction requested=true 2023-04-01 14:54:30,416 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:30,416 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:30,416 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:30,416 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:30,416 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:30,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:30,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:30,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:30,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:30,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59563835[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:30,420 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:30,420 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:30,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:30,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:30,423 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,423 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:30,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:30,424 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:30,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:30,426 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:30,427 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:30,427 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:30,427 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:30,427 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:30,428 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:30,428 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:30,428 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:30,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:30,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:30,430 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:30,430 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,433 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:30,433 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:30,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:30,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:30,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:30,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:30,433 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:30,434 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:30,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,434 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:30,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:30,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:30,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:30,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:30,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:30,534 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:30,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:30,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:30,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:30,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:30,541 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:30,541 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:30,541 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:30,541 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:30,541 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:30,541 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:30,543 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:30,543 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:30,543 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:30,544 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:30,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:30,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:30,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:30,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:30,545 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:30,545 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. started... 2023-04-01 14:54:30,546 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-01 14:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:30,546 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:30,562 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040138d8e79b411242bbaf6eb5c4adceff01 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040138d8e79b411242bbaf6eb5c4adceff01 2023-04-01 14:54:30,568 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=373, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/e5f9d1266d6e482dbb33e12ea8d039bc 2023-04-01 14:54:30,582 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/e7650a1b538d4c44b750eafe587de886 2023-04-01 14:54:30,861 DEBUG [Thread-994] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:60465 2023-04-01 14:54:30,862 DEBUG [Thread-994] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:30,865 DEBUG [Thread-996] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:60465 2023-04-01 14:54:30,865 DEBUG [Thread-996] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:30,866 DEBUG [Thread-1000] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2efdb02f to 127.0.0.1:60465 2023-04-01 14:54:30,866 DEBUG [Thread-1000] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:30,869 DEBUG [Thread-998] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:60465 2023-04-01 14:54:30,869 DEBUG [Thread-998] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:30,994 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/c0867b6cfa754ad7bb87bf237af30797 2023-04-01 14:54:30,998 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/e5f9d1266d6e482dbb33e12ea8d039bc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/e5f9d1266d6e482dbb33e12ea8d039bc 2023-04-01 14:54:31,001 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/e5f9d1266d6e482dbb33e12ea8d039bc, entries=150, sequenceid=373, filesize=25.4 K 2023-04-01 14:54:31,002 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/e7650a1b538d4c44b750eafe587de886 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/e7650a1b538d4c44b750eafe587de886 2023-04-01 14:54:31,006 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/e7650a1b538d4c44b750eafe587de886, entries=150, sequenceid=373, filesize=11.9 K 2023-04-01 14:54:31,006 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/c0867b6cfa754ad7bb87bf237af30797 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/c0867b6cfa754ad7bb87bf237af30797 2023-04-01 14:54:31,010 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/c0867b6cfa754ad7bb87bf237af30797, entries=150, sequenceid=373, filesize=11.9 K 2023-04-01 14:54:31,011 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=0 B/0 for 9a2eee92d9536149526dfe0ebd08db5f in 465ms, sequenceid=373, compaction requested=true 2023-04-01 14:54:31,011 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:31,011 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:31,015 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:31,015 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:31,015 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:31,015 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32f3c7b2[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:31,017 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:31,017 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:31,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:31,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:31,018 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:31,018 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,018 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:31,018 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:31,018 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:31,019 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:31,021 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:31,021 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:31,021 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:31,021 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:31,021 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:31,023 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:31,023 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:31,023 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:31,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:31,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:31,024 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:31,027 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:31,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:31,027 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:31,027 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:31,027 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:31,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:31,027 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:31,027 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:31,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:31,027 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:31,027 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:31,027 DEBUG [Thread-993] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:31,027 DEBUG [Thread-993] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:31,127 DEBUG [Thread-993] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:31,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:31,346 DEBUG [Thread-983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21281a37 to 127.0.0.1:60465 2023-04-01 14:54:31,346 DEBUG [Thread-983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:31,352 DEBUG [Thread-989] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:60465 2023-04-01 14:54:31,352 DEBUG [Thread-989] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:31,676 DEBUG [Thread-987] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:60465 2023-04-01 14:54:31,676 DEBUG [Thread-987] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:31,709 DEBUG [Thread-991] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:60465 2023-04-01 14:54:31,709 DEBUG [Thread-991] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:31,730 DEBUG [Thread-985] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:60465 2023-04-01 14:54:31,730 DEBUG [Thread-985] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:31,730 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 14:54:31,730 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 130 2023-04-01 14:54:31,730 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 16 2023-04-01 14:54:31,730 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 18 2023-04-01 14:54:31,731 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 128 2023-04-01 14:54:31,731 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-04-01 14:54:31,731 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 14:54:31,731 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4779 2023-04-01 14:54:31,731 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4875 2023-04-01 14:54:31,731 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 14:54:31,731 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 2013 2023-04-01 14:54:31,731 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 6039 rows 2023-04-01 14:54:31,731 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 2016 2023-04-01 14:54:31,731 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 6048 rows 2023-04-01 14:54:31,731 INFO [Listener at localhost/46667] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 14:54:31,731 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7829b43e to 127.0.0.1:60465 2023-04-01 14:54:31,731 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:31,733 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 14:54:31,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-01 14:54:31,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:31,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 14:54:31,738 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360871738"}]},"ts":"1680360871738"} 2023-04-01 14:54:31,739 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 14:54:31,741 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 14:54:31,742 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2eee92d9536149526dfe0ebd08db5f, UNASSIGN}] 2023-04-01 14:54:31,743 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2eee92d9536149526dfe0ebd08db5f, UNASSIGN 2023-04-01 14:54:31,744 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=9a2eee92d9536149526dfe0ebd08db5f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:31,744 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360871744"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360871744"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360871744"}]},"ts":"1680360871744"} 2023-04-01 14:54:31,745 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:54:31,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 14:54:31,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:31,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9a2eee92d9536149526dfe0ebd08db5f, disabling compactions & flushes 2023-04-01 14:54:31,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:31,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:31,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. after waiting 0 ms 2023-04-01 14:54:31,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:31,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 9a2eee92d9536149526dfe0ebd08db5f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-01 14:54:31,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=A 2023-04-01 14:54:31,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:31,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=B 2023-04-01 14:54:31,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:31,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2eee92d9536149526dfe0ebd08db5f, store=C 2023-04-01 14:54:31,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:32,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 14:54:32,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401b6bf7b5f819042e7b85433affadee91c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b6bf7b5f819042e7b85433affadee91c 2023-04-01 14:54:32,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 14:54:32,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=381, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/dc7bba71bb0e495ca223aaf79d4d53bf 2023-04-01 14:54:32,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 14:54:33,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/20dbc3f57e4f478c8789bcb2eff26e6e 2023-04-01 14:54:33,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/189f5bec84864203a5dc1a432c208e8d 2023-04-01 14:54:33,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/A/dc7bba71bb0e495ca223aaf79d4d53bf as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/dc7bba71bb0e495ca223aaf79d4d53bf 2023-04-01 14:54:33,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/dc7bba71bb0e495ca223aaf79d4d53bf, entries=150, sequenceid=381, filesize=25.4 K 2023-04-01 14:54:33,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/B/20dbc3f57e4f478c8789bcb2eff26e6e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/20dbc3f57e4f478c8789bcb2eff26e6e 2023-04-01 14:54:33,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/20dbc3f57e4f478c8789bcb2eff26e6e, entries=150, sequenceid=381, filesize=11.9 K 2023-04-01 14:54:33,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/.tmp/C/189f5bec84864203a5dc1a432c208e8d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/189f5bec84864203a5dc1a432c208e8d 2023-04-01 14:54:33,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/189f5bec84864203a5dc1a432c208e8d, entries=150, sequenceid=381, filesize=11.9 K 2023-04-01 14:54:33,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 9a2eee92d9536149526dfe0ebd08db5f in 1654ms, sequenceid=381, compaction requested=true 2023-04-01 14:54:33,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/266a372cef5b4350ace50fef50662e99, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3620e2be25b6424ba0cb1266bca5e984, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/869835375fbb458c9c53e2768a0b206a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/59ec64728a2c44fe84cdf713da2f5f38, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6d33cfd3a03d45c1a995ea4cd63e89c7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/0d4e9b459e11446e956e5b6a7f8730b4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/09d8a91723ac4340a6175fbbda7285d2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/284c1388517845bc94929d808a7537a8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/4cfa12a2947041f8b0e5de625ba76f7a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6b7dae358867407f94767316817401d5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/494462fb51eb41c9b56098c16a56ecef, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3e94debeb4b344239e5ac976083800c2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/517afc3d40d0421d8e326a2b3bc3e552, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/d08a532a046447a1b7e5176c2ba5bfe5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c1f9d12c218b4564aba3a1efb50a0285, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c34fd9d97f144fcbadfc0d3bfdd4d76a] to archive 2023-04-01 14:54:33,566 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:54:33,567 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/266a372cef5b4350ace50fef50662e99 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/266a372cef5b4350ace50fef50662e99 2023-04-01 14:54:33,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3620e2be25b6424ba0cb1266bca5e984 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3620e2be25b6424ba0cb1266bca5e984 2023-04-01 14:54:33,570 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/869835375fbb458c9c53e2768a0b206a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/869835375fbb458c9c53e2768a0b206a 2023-04-01 14:54:33,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/59ec64728a2c44fe84cdf713da2f5f38 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/59ec64728a2c44fe84cdf713da2f5f38 2023-04-01 14:54:33,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6d33cfd3a03d45c1a995ea4cd63e89c7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6d33cfd3a03d45c1a995ea4cd63e89c7 2023-04-01 14:54:33,573 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/0d4e9b459e11446e956e5b6a7f8730b4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/0d4e9b459e11446e956e5b6a7f8730b4 2023-04-01 14:54:33,574 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/09d8a91723ac4340a6175fbbda7285d2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/09d8a91723ac4340a6175fbbda7285d2 2023-04-01 14:54:33,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/284c1388517845bc94929d808a7537a8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/284c1388517845bc94929d808a7537a8 2023-04-01 14:54:33,576 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/4cfa12a2947041f8b0e5de625ba76f7a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/4cfa12a2947041f8b0e5de625ba76f7a 2023-04-01 14:54:33,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6b7dae358867407f94767316817401d5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/6b7dae358867407f94767316817401d5 2023-04-01 14:54:33,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/494462fb51eb41c9b56098c16a56ecef to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/494462fb51eb41c9b56098c16a56ecef 2023-04-01 14:54:33,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3e94debeb4b344239e5ac976083800c2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/3e94debeb4b344239e5ac976083800c2 2023-04-01 14:54:33,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/517afc3d40d0421d8e326a2b3bc3e552 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/517afc3d40d0421d8e326a2b3bc3e552 2023-04-01 14:54:33,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/d08a532a046447a1b7e5176c2ba5bfe5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/d08a532a046447a1b7e5176c2ba5bfe5 2023-04-01 14:54:33,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c1f9d12c218b4564aba3a1efb50a0285 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c1f9d12c218b4564aba3a1efb50a0285 2023-04-01 14:54:33,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c34fd9d97f144fcbadfc0d3bfdd4d76a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/c34fd9d97f144fcbadfc0d3bfdd4d76a 2023-04-01 14:54:33,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/ac5563f060ba4f3599097004cc3c78cb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cbbb03ffc372435f99d95e396a9eb0e4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/3805a3dbb3c24dcab5c8dcd3479b0a30, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/f927000734d14f1aa61656211fdb6023, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/9a8d21704c6a48a7bd9ef0d037d9ed0b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/38dcac3fe4c64b51a7b7707a51d3aab7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/711da8eb02a64d76aab06da0a73bc1b6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1484216803a749f28152105dc8475586, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cc4e9727772b432782e13c23e2018c00, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/82b9020c310641ce9cbc5e0eb5c8e986, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/84a22b4bc7e74a5491e1b8d5bf2db7d1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/380f62aab1074e0aaeb2105266f36a9e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/2ec96a8763944939a083d395c0a7a3e1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1fcc4686908549dcae27de30678deaeb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/a84f2a332032465fa3f9d51f08217a98, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1d417e3946474e03821e8d32d0c38ed2] to archive 2023-04-01 14:54:33,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:54:33,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/ac5563f060ba4f3599097004cc3c78cb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/ac5563f060ba4f3599097004cc3c78cb 2023-04-01 14:54:33,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cbbb03ffc372435f99d95e396a9eb0e4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cbbb03ffc372435f99d95e396a9eb0e4 2023-04-01 14:54:33,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/3805a3dbb3c24dcab5c8dcd3479b0a30 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/3805a3dbb3c24dcab5c8dcd3479b0a30 2023-04-01 14:54:33,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/f927000734d14f1aa61656211fdb6023 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/f927000734d14f1aa61656211fdb6023 2023-04-01 14:54:33,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/9a8d21704c6a48a7bd9ef0d037d9ed0b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/9a8d21704c6a48a7bd9ef0d037d9ed0b 2023-04-01 14:54:33,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/38dcac3fe4c64b51a7b7707a51d3aab7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/38dcac3fe4c64b51a7b7707a51d3aab7 2023-04-01 14:54:33,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/711da8eb02a64d76aab06da0a73bc1b6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/711da8eb02a64d76aab06da0a73bc1b6 2023-04-01 14:54:33,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1484216803a749f28152105dc8475586 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1484216803a749f28152105dc8475586 2023-04-01 14:54:33,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cc4e9727772b432782e13c23e2018c00 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/cc4e9727772b432782e13c23e2018c00 2023-04-01 14:54:33,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/82b9020c310641ce9cbc5e0eb5c8e986 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/82b9020c310641ce9cbc5e0eb5c8e986 2023-04-01 14:54:33,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/84a22b4bc7e74a5491e1b8d5bf2db7d1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/84a22b4bc7e74a5491e1b8d5bf2db7d1 2023-04-01 14:54:33,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/380f62aab1074e0aaeb2105266f36a9e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/380f62aab1074e0aaeb2105266f36a9e 2023-04-01 14:54:33,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/2ec96a8763944939a083d395c0a7a3e1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/2ec96a8763944939a083d395c0a7a3e1 2023-04-01 14:54:33,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1fcc4686908549dcae27de30678deaeb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1fcc4686908549dcae27de30678deaeb 2023-04-01 14:54:33,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/a84f2a332032465fa3f9d51f08217a98 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/a84f2a332032465fa3f9d51f08217a98 2023-04-01 14:54:33,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1d417e3946474e03821e8d32d0c38ed2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/1d417e3946474e03821e8d32d0c38ed2 2023-04-01 14:54:33,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/326c2c194b7d45ed8a32a2d522b15cea, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f44a229727614242b547b35565e2eafc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/fb37408696d842b9a53ccfb97e36e50e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/02134c5d9a6a444f9072dd227ddaab50, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/9c213f1137ff444dbc863028c1d4d59b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/31d669309a314ded915c421c30096031, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/e1a9216215834207b4f5eb0b45672f0a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/6d6d55198c024bf69c2320b690849387, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/1ef9d789ea5e415186286908276e731f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/36c323f0acba4bde8b5ce6570722e177, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/3600c12b421f488583ce5aec272eda9d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/3c42b5e9fd0d47c58d088ddaf9953915, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/17fe10ed4272433b83c58e09fd0b4bb9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f47d8c5c9ff54232a59cb68ce8b8e28b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/0fc9d1a87b084e118ca502a25dc1cc0d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/d404c9aa8a224651a52bc2aa57a51208] to archive 2023-04-01 14:54:33,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:54:33,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/326c2c194b7d45ed8a32a2d522b15cea to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/326c2c194b7d45ed8a32a2d522b15cea 2023-04-01 14:54:33,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f44a229727614242b547b35565e2eafc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f44a229727614242b547b35565e2eafc 2023-04-01 14:54:33,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/fb37408696d842b9a53ccfb97e36e50e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/fb37408696d842b9a53ccfb97e36e50e 2023-04-01 14:54:33,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/02134c5d9a6a444f9072dd227ddaab50 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/02134c5d9a6a444f9072dd227ddaab50 2023-04-01 14:54:33,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/9c213f1137ff444dbc863028c1d4d59b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/9c213f1137ff444dbc863028c1d4d59b 2023-04-01 14:54:33,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/31d669309a314ded915c421c30096031 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/31d669309a314ded915c421c30096031 2023-04-01 14:54:33,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/e1a9216215834207b4f5eb0b45672f0a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/e1a9216215834207b4f5eb0b45672f0a 2023-04-01 14:54:33,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/6d6d55198c024bf69c2320b690849387 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/6d6d55198c024bf69c2320b690849387 2023-04-01 14:54:33,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/1ef9d789ea5e415186286908276e731f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/1ef9d789ea5e415186286908276e731f 2023-04-01 14:54:33,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/36c323f0acba4bde8b5ce6570722e177 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/36c323f0acba4bde8b5ce6570722e177 2023-04-01 14:54:33,623 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/3600c12b421f488583ce5aec272eda9d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/3600c12b421f488583ce5aec272eda9d 2023-04-01 14:54:33,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/3c42b5e9fd0d47c58d088ddaf9953915 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/3c42b5e9fd0d47c58d088ddaf9953915 2023-04-01 14:54:33,625 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/17fe10ed4272433b83c58e09fd0b4bb9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/17fe10ed4272433b83c58e09fd0b4bb9 2023-04-01 14:54:33,626 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f47d8c5c9ff54232a59cb68ce8b8e28b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/f47d8c5c9ff54232a59cb68ce8b8e28b 2023-04-01 14:54:33,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/0fc9d1a87b084e118ca502a25dc1cc0d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/0fc9d1a87b084e118ca502a25dc1cc0d 2023-04-01 14:54:33,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/d404c9aa8a224651a52bc2aa57a51208 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/d404c9aa8a224651a52bc2aa57a51208 2023-04-01 14:54:33,631 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/recovered.edits/384.seqid, newMaxSeqId=384, maxSeqId=4 2023-04-01 14:54:33,633 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f. 2023-04-01 14:54:33,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9a2eee92d9536149526dfe0ebd08db5f: 2023-04-01 14:54:33,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:33,634 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=9a2eee92d9536149526dfe0ebd08db5f, regionState=CLOSED 2023-04-01 14:54:33,635 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360873634"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360873634"}]},"ts":"1680360873634"} 2023-04-01 14:54:33,637 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-01 14:54:33,637 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 9a2eee92d9536149526dfe0ebd08db5f, server=jenkins-hbase4.apache.org,40863,1680360818567 in 1.8900 sec 2023-04-01 14:54:33,638 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-01 14:54:33,638 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2eee92d9536149526dfe0ebd08db5f, UNASSIGN in 1.8950 sec 2023-04-01 14:54:33,639 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360873638"}]},"ts":"1680360873638"} 2023-04-01 14:54:33,639 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 14:54:33,641 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 14:54:33,642 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9070 sec 2023-04-01 14:54:33,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-01 14:54:33,842 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-01 14:54:33,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-01 14:54:33,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:33,845 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:33,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-01 14:54:33,846 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:33,851 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:33,853 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/recovered.edits] 2023-04-01 14:54:33,855 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/062fd4cae1444299a5ac8f7a246c9a45 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/062fd4cae1444299a5ac8f7a246c9a45 2023-04-01 14:54:33,856 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/7d4d62bc978148f3bf897211ca7a9d65 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/7d4d62bc978148f3bf897211ca7a9d65 2023-04-01 14:54:33,857 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/8dc9502050be4120ab6633f840133692 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/8dc9502050be4120ab6633f840133692 2023-04-01 14:54:33,858 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/d2750d79f13548f8a91aa590097c9091 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/d2750d79f13548f8a91aa590097c9091 2023-04-01 14:54:33,859 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/dc7bba71bb0e495ca223aaf79d4d53bf to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/dc7bba71bb0e495ca223aaf79d4d53bf 2023-04-01 14:54:33,861 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/e5f9d1266d6e482dbb33e12ea8d039bc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/A/e5f9d1266d6e482dbb33e12ea8d039bc 2023-04-01 14:54:33,863 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/12caac5a064d4efcb7ab35927d576638 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/12caac5a064d4efcb7ab35927d576638 2023-04-01 14:54:33,864 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/20dbc3f57e4f478c8789bcb2eff26e6e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/20dbc3f57e4f478c8789bcb2eff26e6e 2023-04-01 14:54:33,865 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/b2169ecdf15e4e16867abb13497dab85 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/b2169ecdf15e4e16867abb13497dab85 2023-04-01 14:54:33,866 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/bee07b3df9f84e42a06501863d56cbc0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/bee07b3df9f84e42a06501863d56cbc0 2023-04-01 14:54:33,867 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/c5be4369e8f04f058a4888185d2936d0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/c5be4369e8f04f058a4888185d2936d0 2023-04-01 14:54:33,868 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/e7650a1b538d4c44b750eafe587de886 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/B/e7650a1b538d4c44b750eafe587de886 2023-04-01 14:54:33,870 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/0abc8fda0d23467581d295986e34ed94 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/0abc8fda0d23467581d295986e34ed94 2023-04-01 14:54:33,871 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/189f5bec84864203a5dc1a432c208e8d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/189f5bec84864203a5dc1a432c208e8d 2023-04-01 14:54:33,872 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/2a448917fcc94645a36f85bf2620ed49 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/2a448917fcc94645a36f85bf2620ed49 2023-04-01 14:54:33,873 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/4bdcc450ef3a4ca7b6853f089de4624e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/4bdcc450ef3a4ca7b6853f089de4624e 2023-04-01 14:54:33,874 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/84cb9f16078541149f094d7c7751ba9b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/84cb9f16078541149f094d7c7751ba9b 2023-04-01 14:54:33,875 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/c0867b6cfa754ad7bb87bf237af30797 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/C/c0867b6cfa754ad7bb87bf237af30797 2023-04-01 14:54:33,878 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/recovered.edits/384.seqid to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f/recovered.edits/384.seqid 2023-04-01 14:54:33,878 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/9a2eee92d9536149526dfe0ebd08db5f 2023-04-01 14:54:33,878 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:54:33,879 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 14:54:33,879 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-01 14:54:33,882 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401013f2e556a564d93afa54cb4852afb88 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401013f2e556a564d93afa54cb4852afb88 2023-04-01 14:54:33,883 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011cb1ececabe546f1ae805779d07155e1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011cb1ececabe546f1ae805779d07155e1 2023-04-01 14:54:33,884 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040138d8e79b411242bbaf6eb5c4adceff01 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040138d8e79b411242bbaf6eb5c4adceff01 2023-04-01 14:54:33,885 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013ae319efece940e6bc2d2cc56635907a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013ae319efece940e6bc2d2cc56635907a 2023-04-01 14:54:33,886 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040153aa691909be4c8f907c69b899773034 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040153aa691909be4c8f907c69b899773034 2023-04-01 14:54:33,886 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040165eaf9c0d5d3480aa7eb58cbd20b986e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040165eaf9c0d5d3480aa7eb58cbd20b986e 2023-04-01 14:54:33,887 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304016e5c47140d7d40d0b7f27a54483d59bc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304016e5c47140d7d40d0b7f27a54483d59bc 2023-04-01 14:54:33,888 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017911ad2c2a714f508ee4e8fbc51a4a64 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304017911ad2c2a714f508ee4e8fbc51a4a64 2023-04-01 14:54:33,889 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040181207421cd6e4528bfd2377ba5d28e03 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040181207421cd6e4528bfd2377ba5d28e03 2023-04-01 14:54:33,890 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401859631f2e46a41238e34bb0ed6bc4593 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401859631f2e46a41238e34bb0ed6bc4593 2023-04-01 14:54:33,891 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018d41936308c04235a2f9aede0b51dbd9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018d41936308c04235a2f9aede0b51dbd9 2023-04-01 14:54:33,893 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b0b601e4117b41aaa84bbb785c3cb6c5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b0b601e4117b41aaa84bbb785c3cb6c5 2023-04-01 14:54:33,894 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b603db06c9184b2b8a0c87d74eccaa88 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b603db06c9184b2b8a0c87d74eccaa88 2023-04-01 14:54:33,895 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b6bf7b5f819042e7b85433affadee91c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b6bf7b5f819042e7b85433affadee91c 2023-04-01 14:54:33,896 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ce3f50ef2b174e508d62da291dc0dbb7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ce3f50ef2b174e508d62da291dc0dbb7 2023-04-01 14:54:33,897 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e16d16988862464a8558c21af21bd2bf to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e16d16988862464a8558c21af21bd2bf 2023-04-01 14:54:33,898 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e69525ff3d66434a83067e84ccca0c4b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e69525ff3d66434a83067e84ccca0c4b 2023-04-01 14:54:33,899 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ea0fa4d3af2c40e793d0d1fc62bb543f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ea0fa4d3af2c40e793d0d1fc62bb543f 2023-04-01 14:54:33,900 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ebd46c26e5854e5996bf87db01941e0a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ebd46c26e5854e5996bf87db01941e0a 2023-04-01 14:54:33,901 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 14:54:33,903 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:33,905 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 14:54:33,907 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 14:54:33,908 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:33,908 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 14:54:33,908 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680360873908"}]},"ts":"9223372036854775807"} 2023-04-01 14:54:33,912 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 14:54:33,912 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9a2eee92d9536149526dfe0ebd08db5f, NAME => 'TestAcidGuarantees,,1680360848122.9a2eee92d9536149526dfe0ebd08db5f.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 14:54:33,913 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 14:54:33,913 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680360873913"}]},"ts":"9223372036854775807"} 2023-04-01 14:54:33,914 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 14:54:33,916 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:33,917 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 73 msec 2023-04-01 14:54:33,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-01 14:54:33,947 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-01 14:54:33,959 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=239 (was 237) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-620753973_15 at /127.0.0.1:54432 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f5d57d6-shared-pool-38 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/cluster_9b9f408e-8fe1-e5f9-3143-13e52c404ab9/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f5d57d6-shared-pool-37 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-620753973_15 at /127.0.0.1:54468 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f5d57d6-shared-pool-36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/cluster_9b9f408e-8fe1-e5f9-3143-13e52c404ab9/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f5d57d6-shared-pool-35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-620753973_15 at /127.0.0.1:41804 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=552 (was 586), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=395 (was 373) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3342 (was 3564) 2023-04-01 14:54:33,971 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=239, OpenFileDescriptor=552, MaxFileDescriptor=60000, SystemLoadAverage=395, ProcessCount=169, AvailableMemoryMB=3342 2023-04-01 14:54:33,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 14:54:33,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 14:54:33,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 14:54:33,975 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 14:54:33,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-01 14:54:33,976 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 14:54:33,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-01 14:54:33,977 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:33,977 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d empty. 2023-04-01 14:54:33,978 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:33,978 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:54:34,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-01 14:54:34,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-01 14:54:34,385 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 14:54:34,386 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5326183795415453ea78c16f1f182c7d, NAME => 'TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp 2023-04-01 14:54:34,391 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:54:34,392 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5326183795415453ea78c16f1f182c7d, disabling compactions & flushes 2023-04-01 14:54:34,392 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:34,392 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:34,392 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. after waiting 0 ms 2023-04-01 14:54:34,392 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:34,392 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:34,392 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:34,394 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 14:54:34,395 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360874395"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360874395"}]},"ts":"1680360874395"} 2023-04-01 14:54:34,396 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 14:54:34,396 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 14:54:34,396 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360874396"}]},"ts":"1680360874396"} 2023-04-01 14:54:34,397 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 14:54:34,400 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5326183795415453ea78c16f1f182c7d, ASSIGN}] 2023-04-01 14:54:34,402 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5326183795415453ea78c16f1f182c7d, ASSIGN 2023-04-01 14:54:34,402 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5326183795415453ea78c16f1f182c7d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40863,1680360818567; forceNewPlan=false, retain=false 2023-04-01 14:54:34,553 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=5326183795415453ea78c16f1f182c7d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:34,554 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360874553"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360874553"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360874553"}]},"ts":"1680360874553"} 2023-04-01 14:54:34,555 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:54:34,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-01 14:54:34,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:34,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5326183795415453ea78c16f1f182c7d, NAME => 'TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:54:34,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:34,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:54:34,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:34,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:34,712 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:34,713 DEBUG [StoreOpener-5326183795415453ea78c16f1f182c7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A 2023-04-01 14:54:34,713 DEBUG [StoreOpener-5326183795415453ea78c16f1f182c7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A 2023-04-01 14:54:34,713 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:54:34,713 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5326183795415453ea78c16f1f182c7d columnFamilyName A 2023-04-01 14:54:34,714 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] regionserver.HStore(310): Store=5326183795415453ea78c16f1f182c7d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:54:34,714 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:34,715 DEBUG [StoreOpener-5326183795415453ea78c16f1f182c7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B 2023-04-01 14:54:34,715 DEBUG [StoreOpener-5326183795415453ea78c16f1f182c7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B 2023-04-01 14:54:34,715 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:54:34,716 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5326183795415453ea78c16f1f182c7d columnFamilyName B 2023-04-01 14:54:34,716 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] regionserver.HStore(310): Store=5326183795415453ea78c16f1f182c7d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:54:34,716 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:34,717 DEBUG [StoreOpener-5326183795415453ea78c16f1f182c7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C 2023-04-01 14:54:34,717 DEBUG [StoreOpener-5326183795415453ea78c16f1f182c7d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C 2023-04-01 14:54:34,717 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:54:34,718 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5326183795415453ea78c16f1f182c7d columnFamilyName C 2023-04-01 14:54:34,718 INFO [StoreOpener-5326183795415453ea78c16f1f182c7d-1] regionserver.HStore(310): Store=5326183795415453ea78c16f1f182c7d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:54:34,718 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:34,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:34,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:34,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 14:54:34,721 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:34,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 14:54:34,723 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5326183795415453ea78c16f1f182c7d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62032034, jitterRate=-0.0756506621837616}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 14:54:34,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:34,723 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., pid=30, masterSystemTime=1680360874706 2023-04-01 14:54:34,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:34,725 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:34,725 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=5326183795415453ea78c16f1f182c7d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:34,725 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360874725"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680360874725"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680360874725"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680360874725"}]},"ts":"1680360874725"} 2023-04-01 14:54:34,727 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-01 14:54:34,727 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 in 171 msec 2023-04-01 14:54:34,728 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-01 14:54:34,729 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5326183795415453ea78c16f1f182c7d, ASSIGN in 327 msec 2023-04-01 14:54:34,729 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 14:54:34,729 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360874729"}]},"ts":"1680360874729"} 2023-04-01 14:54:34,730 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 14:54:34,732 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 14:54:34,733 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 759 msec 2023-04-01 14:54:35,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-01 14:54:35,079 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-01 14:54:35,080 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:35,085 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:35,086 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:35,087 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:35,088 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 14:54:35,090 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57246, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 14:54:35,091 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:35,095 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:35,095 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:35,102 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:35,102 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:35,106 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:35,106 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x7810a231 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:35,110 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@104dbf4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:35,110 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x08cd7cb0 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:35,114 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@753156b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:35,114 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:35,120 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:35,120 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:35,124 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:35,125 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:35,130 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:35,131 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x2de064cb to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:35,135 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f8ff99f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:35,135 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x54a39be8 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:54:35,140 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a4b7b8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:54:35,148 DEBUG [hconnection-0x24cc54fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:35,149 DEBUG [hconnection-0x784fd50a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:35,149 DEBUG [hconnection-0x46c65493-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:35,149 DEBUG [hconnection-0x44c79d72-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:35,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:35,152 DEBUG [hconnection-0x42b68ec2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:35,152 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60084, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:35,152 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:35,151 DEBUG [hconnection-0x1c58b04d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:35,154 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60112, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:35,153 DEBUG [hconnection-0x3ef98b30-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:35,150 DEBUG [hconnection-0x54dd162a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:35,150 DEBUG [hconnection-0x29789a8a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:35,154 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60090, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:35,155 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60126, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:35,152 DEBUG [hconnection-0x297cf8eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:54:35,156 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:35,158 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:35,159 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60132, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:35,160 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60138, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:35,160 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60152, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:54:35,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:35,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:35,162 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:35,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:35,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:35,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:35,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:35,169 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,169 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-01 14:54:35,169 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:35,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:35,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:35,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:35,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:35,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:35,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:35,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:35,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:35,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:35,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:35,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:35,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:35,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:35,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:35,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:35,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:35,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:35,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:35,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:35,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:35,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:35,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:35,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:35,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:35,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,185 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,186 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,186 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:35,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@470f262f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:35,188 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:35,188 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,189 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:35,189 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:35,189 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,190 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:35,189 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:35,192 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360935189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360875181, max timestamp=1680360875190Num uniques 859; 2023-04-01 14:54:35,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:35,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:35,195 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:35,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:35,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:35,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:35,195 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:35,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680360875181, max timestamp=1680360875190Num uniques 859; 2023-04-01 14:54:35,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 14:54:35,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680360875181, max timestamp=1680360875190Num uniques 859; 2023-04-01 14:54:35,197 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,197 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:35,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-01 14:54:35,197 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,198 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:35,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:35,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,199 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:35,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:35,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:35,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:35,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:35,202 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:35,202 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:35,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:35,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:35,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:35,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360935211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360935212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360935212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360935213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360935296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,302 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:35,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:35,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:35,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:35,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:35,307 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:35,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:35,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:35,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:35,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,308 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,309 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,309 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:35,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:35,309 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:35,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:35,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:35,311 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:35,312 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:35,312 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:35,312 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:35,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:35,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:35,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:35,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:35,317 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360935317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:35,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ba6fcbc[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:35,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:35,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,320 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,320 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:35,321 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:35,321 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:35,321 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,321 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:35,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:35,323 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360935322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360935322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360935322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:35,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:35,326 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:35,326 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:35,326 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:35,326 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:35,326 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:35,328 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,329 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,330 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,330 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:35,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,335 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,335 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,335 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,335 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,335 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,336 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,336 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,336 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:35,336 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:35,336 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:35,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:35,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:35,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:35,336 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:35,336 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:35,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:35,436 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:35,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:35,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:35,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:35,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:35,439 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:35,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:35,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:35,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:35,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,441 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,441 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:35,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:35,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:35,441 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,441 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:35,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:35,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:35,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:35,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:35,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:35,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:35,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:35,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:35,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:35,443 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:35,443 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:35,443 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:35,443 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:35,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:35,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:35,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:35,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:35,445 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,445 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,445 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,445 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:35,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:35,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2359f26d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:35,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,449 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,449 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:35,449 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:35,449 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:35,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:35,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,450 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:35,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:35,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:35,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:35,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:35,453 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:35,453 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:35,453 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:35,453 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:35,453 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:35,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:35,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:35,456 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:35,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,460 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:35,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:35,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:35,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:35,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:35,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:35,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:35,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,467 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:35,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,468 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:35,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:35,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:35,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360935497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360935519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360935525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360935525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360935526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,568 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:35,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:35,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:35,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:35,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:35,571 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:35,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:35,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:35,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:35,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,575 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:35,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:35,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:35,576 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,576 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:35,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:35,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:35,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:35,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:35,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:35,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:35,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:35,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:35,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:35,577 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:35,577 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:35,577 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:35,577 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:35,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:35,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:35,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:35,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:35,579 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,579 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,579 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,579 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:35,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:35,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f3b6fe3[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:35,581 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:35,581 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,582 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:35,582 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:35,582 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:35,582 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,582 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:35,583 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:35,583 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,583 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:35,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,584 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:35,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:35,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:35,586 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:35,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:35,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:35,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:35,586 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:35,588 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,588 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,588 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:35,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:35,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:35,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:35,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:35,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:35,589 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:35,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,592 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:35,592 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:35,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:35,593 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,593 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:35,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:35,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:35,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:35,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:35,593 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,593 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,593 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,593 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,593 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:35,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,594 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:35,594 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:35,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/383b832b887843748d036532638ced0e 2023-04-01 14:54:35,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/6b2823765b274968917ed174e52b48fe 2023-04-01 14:54:35,634 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/4dea448c9ec349118dd5aa20b02d98d4 2023-04-01 14:54:35,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/383b832b887843748d036532638ced0e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/383b832b887843748d036532638ced0e 2023-04-01 14:54:35,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/383b832b887843748d036532638ced0e, entries=150, sequenceid=12, filesize=11.6 K 2023-04-01 14:54:35,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/6b2823765b274968917ed174e52b48fe as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/6b2823765b274968917ed174e52b48fe 2023-04-01 14:54:35,647 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/6b2823765b274968917ed174e52b48fe, entries=150, sequenceid=12, filesize=11.6 K 2023-04-01 14:54:35,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/4dea448c9ec349118dd5aa20b02d98d4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4dea448c9ec349118dd5aa20b02d98d4 2023-04-01 14:54:35,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4dea448c9ec349118dd5aa20b02d98d4, entries=150, sequenceid=12, filesize=11.6 K 2023-04-01 14:54:35,652 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.68 KB/129720 for 5326183795415453ea78c16f1f182c7d in 483ms, sequenceid=12, compaction requested=false 2023-04-01 14:54:35,652 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-01 14:54:35,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:35,694 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:35,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:35,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:35,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:35,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:35,697 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:35,697 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:35,697 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:35,697 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:35,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,699 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,699 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:35,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:35,699 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,699 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:35,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:35,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:35,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:35,701 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:35,701 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:35,701 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:35,701 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:35,701 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:35,701 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:35,701 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:35,702 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=126.68 KB heapSize=363.23 KB 2023-04-01 14:54:35,702 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:35,702 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:35,702 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:35,702 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:35,702 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:35,702 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:35,713 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/c6fe2920e96645dea42dfcb826a519b6 2023-04-01 14:54:35,726 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/9c2406f728a24738849762a1f18f05de 2023-04-01 14:54:35,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:35,801 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:35,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360935820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360935823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360935827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360935828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360935829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:35,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:35,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360935922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360936124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,138 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/785a0bea67ea478fa68341376020c2b0 2023-04-01 14:54:36,143 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/c6fe2920e96645dea42dfcb826a519b6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/c6fe2920e96645dea42dfcb826a519b6 2023-04-01 14:54:36,147 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/c6fe2920e96645dea42dfcb826a519b6, entries=150, sequenceid=48, filesize=11.6 K 2023-04-01 14:54:36,147 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/9c2406f728a24738849762a1f18f05de as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/9c2406f728a24738849762a1f18f05de 2023-04-01 14:54:36,151 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/9c2406f728a24738849762a1f18f05de, entries=150, sequenceid=48, filesize=11.6 K 2023-04-01 14:54:36,152 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/785a0bea67ea478fa68341376020c2b0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/785a0bea67ea478fa68341376020c2b0 2023-04-01 14:54:36,155 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/785a0bea67ea478fa68341376020c2b0, entries=150, sequenceid=48, filesize=11.6 K 2023-04-01 14:54:36,156 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~126.68 KB/129720, heapSize ~363.19 KB/371904, currentSize=53.67 KB/54960 for 5326183795415453ea78c16f1f182c7d in 454ms, sequenceid=48, compaction requested=false 2023-04-01 14:54:36,157 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:36,157 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,157 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:36,157 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:36,157 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,157 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:36,157 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:36,158 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,158 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,159 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,159 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:36,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@313b1e75[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:36,160 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:36,160 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,162 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:36,162 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:36,162 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:36,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:36,167 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:36,167 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:36,167 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:36,176 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:36,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:36,185 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:36,185 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:36,185 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:36,185 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:36,185 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:36,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:36,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:36,188 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,189 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,192 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,192 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,192 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:36,192 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:36,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:36,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:36,192 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:36,192 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:36,192 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,192 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:36,192 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,192 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:36,192 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:36,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:36,192 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:36,293 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:36,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:36,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:36,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:36,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:36,296 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:36,296 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:36,296 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:36,296 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:36,297 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,297 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:36,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:36,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:36,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:36,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:36,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:36,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:36,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:36,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:36,301 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,301 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:36,301 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-01 14:54:36,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:36,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:36,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:36,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:36,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:36,302 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:36,313 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/82f8942348f74ed6b93382fe339257e2 2023-04-01 14:54:36,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:36,330 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:36,338 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/338af4d480bb4ae1bb2f8f186ce782e3 2023-04-01 14:54:36,350 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/539ea0f4c71042bcb5dda63d237faa29 2023-04-01 14:54:36,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360936352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360876345, max timestamp=1680360876354Num uniques 859; 2023-04-01 14:54:36,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:36,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360876345, max timestamp=1680360876354Num uniques 859; 2023-04-01 14:54:36,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:36,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360876345, max timestamp=1680360876354Num uniques 859; 2023-04-01 14:54:36,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:36,357 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/82f8942348f74ed6b93382fe339257e2 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/82f8942348f74ed6b93382fe339257e2 2023-04-01 14:54:36,361 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/82f8942348f74ed6b93382fe339257e2, entries=150, sequenceid=59, filesize=11.6 K 2023-04-01 14:54:36,362 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/338af4d480bb4ae1bb2f8f186ce782e3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/338af4d480bb4ae1bb2f8f186ce782e3 2023-04-01 14:54:36,367 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/338af4d480bb4ae1bb2f8f186ce782e3, entries=150, sequenceid=59, filesize=11.6 K 2023-04-01 14:54:36,368 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/539ea0f4c71042bcb5dda63d237faa29 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/539ea0f4c71042bcb5dda63d237faa29 2023-04-01 14:54:36,372 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/539ea0f4c71042bcb5dda63d237faa29, entries=150, sequenceid=59, filesize=11.6 K 2023-04-01 14:54:36,372 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=87.22 KB/89310 for 5326183795415453ea78c16f1f182c7d in 71ms, sequenceid=59, compaction requested=true 2023-04-01 14:54:36,373 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:36,373 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,373 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:36,373 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:36,373 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,373 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:36,373 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:36,375 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:36,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=100.63 KB heapSize=285.38 KB 2023-04-01 14:54:36,376 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,376 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:36,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:36,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:36,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:36,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:36,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:36,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:36,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:36,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0457f2[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:36,379 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:36,379 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,380 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,380 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,380 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:36,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:36,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:36,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:36,382 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:36,382 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,382 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:36,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,382 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:36,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:36,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:36,387 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:36,387 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:36,387 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:36,387 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:36,387 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:36,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:36,390 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,391 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360936390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:36,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:36,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:36,395 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:36,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:36,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:36,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,396 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,396 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,396 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,396 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:36,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:36,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:36,396 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,396 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:36,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360936394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360936395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360936428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360936458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,496 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:36,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:36,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:36,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360936496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:36,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:36,500 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:36,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:36,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:36,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:36,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360936499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360936499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,501 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:36,502 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:36,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:36,502 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,502 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:36,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:36,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:36,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:36,503 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:36,503 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:36,503 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:36,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:36,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:36,504 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,504 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:36,504 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:36,504 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:36,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:36,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:36,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:36,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,505 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,506 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,506 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:36,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:36,507 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:36,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21221ea5[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:36,507 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,509 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:36,509 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:36,509 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:36,509 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,509 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:36,511 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,511 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:36,512 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:36,512 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:36,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:36,514 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:36,514 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:36,514 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:36,514 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:36,515 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:36,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,517 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:36,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:36,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,518 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,520 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,520 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,520 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,521 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,521 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,521 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:36,521 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:36,521 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:36,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:36,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:36,521 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:36,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:36,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:36,530 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:36,530 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:36,630 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:36,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:36,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:36,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:36,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:36,634 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:36,634 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:36,634 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:36,634 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:36,634 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,634 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,635 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:36,635 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:36,636 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,636 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:36,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:36,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:36,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:36,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:36,637 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:36,637 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:36,637 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:36,637 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:36,637 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:36,637 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,637 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:36,637 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:36,638 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,638 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:36,638 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:36,638 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,638 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:36,638 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:36,640 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,640 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,641 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,641 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:36,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75322915[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:36,642 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:36,642 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,643 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:36,643 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:36,643 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:36,644 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,644 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:36,645 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,645 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,645 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,645 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:36,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:36,647 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:36,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:36,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:36,647 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:36,647 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:36,652 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,652 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,652 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:36,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,653 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,656 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:36,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:36,658 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:36,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:36,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:36,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:36,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:36,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:36,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,659 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:36,659 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:36,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:36,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360936661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360936700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360936702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360936702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,759 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:36,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:36,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:36,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:36,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:36,762 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:36,762 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:36,762 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:36,762 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:36,763 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,763 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,764 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:36,764 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:36,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:36,764 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,764 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:36,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:36,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:36,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:36,765 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:36,765 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:36,765 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:36,766 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:36,766 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:36,766 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,766 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:36,766 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:36,766 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,766 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:36,766 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:36,766 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,766 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:36,766 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:36,768 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:36,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c935970[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:36,769 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:36,770 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:36,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:36,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:36,772 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,772 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:36,773 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,773 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:36,773 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:36,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,773 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:36,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:36,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:36,775 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:36,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:36,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:36,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:36,776 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:36,777 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,777 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,777 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:36,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:36,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:36,778 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,781 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,781 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,781 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,781 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,781 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:36,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:36,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:36,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:36,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:36,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:36,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:36,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:36,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,782 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:36,782 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:36,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:36,799 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/e62a8e99e3674f7390a1021a8dc8b7b8 2023-04-01 14:54:36,817 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/798e210c714e41e68f7382a567dc0f6c 2023-04-01 14:54:36,837 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/e443e1f430af4b7abb623b21a6741327 2023-04-01 14:54:36,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/e62a8e99e3674f7390a1021a8dc8b7b8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e62a8e99e3674f7390a1021a8dc8b7b8 2023-04-01 14:54:36,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e62a8e99e3674f7390a1021a8dc8b7b8, entries=300, sequenceid=92, filesize=18.5 K 2023-04-01 14:54:36,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/798e210c714e41e68f7382a567dc0f6c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/798e210c714e41e68f7382a567dc0f6c 2023-04-01 14:54:36,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/798e210c714e41e68f7382a567dc0f6c, entries=150, sequenceid=92, filesize=11.6 K 2023-04-01 14:54:36,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/e443e1f430af4b7abb623b21a6741327 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e443e1f430af4b7abb623b21a6741327 2023-04-01 14:54:36,856 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e443e1f430af4b7abb623b21a6741327, entries=150, sequenceid=92, filesize=11.6 K 2023-04-01 14:54:36,857 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~305.25 KB/312576, currentSize=73.80 KB/75570 for 5326183795415453ea78c16f1f182c7d in 481ms, sequenceid=92, compaction requested=true 2023-04-01 14:54:36,857 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:36,857 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:36,857 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:54:36,857 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:54:36,857 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:36,857 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:36,858 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54564 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:54:36,859 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/A is initiating minor compaction (all files) 2023-04-01 14:54:36,859 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:54:36,859 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/A in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,859 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/B is initiating minor compaction (all files) 2023-04-01 14:54:36,859 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/383b832b887843748d036532638ced0e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/c6fe2920e96645dea42dfcb826a519b6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/82f8942348f74ed6b93382fe339257e2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e62a8e99e3674f7390a1021a8dc8b7b8] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=53.3 K 2023-04-01 14:54:36,859 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/B in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,859 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/6b2823765b274968917ed174e52b48fe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/9c2406f728a24738849762a1f18f05de, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/338af4d480bb4ae1bb2f8f186ce782e3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/798e210c714e41e68f7382a567dc0f6c] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=46.4 K 2023-04-01 14:54:36,859 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 383b832b887843748d036532638ced0e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680360875164 2023-04-01 14:54:36,860 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6b2823765b274968917ed174e52b48fe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680360875164 2023-04-01 14:54:36,860 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting c6fe2920e96645dea42dfcb826a519b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680360875210 2023-04-01 14:54:36,860 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9c2406f728a24738849762a1f18f05de, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680360875210 2023-04-01 14:54:36,860 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 82f8942348f74ed6b93382fe339257e2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680360875803 2023-04-01 14:54:36,860 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 338af4d480bb4ae1bb2f8f186ce782e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680360875803 2023-04-01 14:54:36,861 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting e62a8e99e3674f7390a1021a8dc8b7b8, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680360876362 2023-04-01 14:54:36,861 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 798e210c714e41e68f7382a567dc0f6c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680360876367 2023-04-01 14:54:36,870 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#B#compaction#210 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:36,871 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#A#compaction#211 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:36,882 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:36,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:36,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:36,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:36,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:36,889 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:36,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:36,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:36,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:36,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,891 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:36,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:36,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:36,896 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,897 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,897 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:36,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:36,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:36,898 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:36,898 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:36,898 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:36,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:36,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:36,900 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:36,900 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:36,900 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:54:36,900 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:36,901 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:36,901 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:36,901 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:36,901 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:36,901 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:36,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:36,935 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:36,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360936969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:36,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:36,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360936970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360937004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360937004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360937006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360937073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360937073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360937276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360937276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,300 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/4b6e5e79fe3f45c18249d843b2118a48 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/4b6e5e79fe3f45c18249d843b2118a48 2023-04-01 14:54:37,305 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5326183795415453ea78c16f1f182c7d/B of 5326183795415453ea78c16f1f182c7d into 4b6e5e79fe3f45c18249d843b2118a48(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:37,305 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:37,306 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/B, priority=12, startTime=1680360876857; duration=0sec 2023-04-01 14:54:37,306 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:37,306 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:54:37,307 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/25fdd2bdc34c4734af9886077274bbf4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/25fdd2bdc34c4734af9886077274bbf4 2023-04-01 14:54:37,307 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:54:37,307 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/C is initiating minor compaction (all files) 2023-04-01 14:54:37,307 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/C in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:37,308 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4dea448c9ec349118dd5aa20b02d98d4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/785a0bea67ea478fa68341376020c2b0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/539ea0f4c71042bcb5dda63d237faa29, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e443e1f430af4b7abb623b21a6741327] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=46.4 K 2023-04-01 14:54:37,308 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 4dea448c9ec349118dd5aa20b02d98d4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680360875164 2023-04-01 14:54:37,308 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 785a0bea67ea478fa68341376020c2b0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680360875210 2023-04-01 14:54:37,309 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 539ea0f4c71042bcb5dda63d237faa29, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680360875803 2023-04-01 14:54:37,309 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e443e1f430af4b7abb623b21a6741327, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680360876367 2023-04-01 14:54:37,311 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5326183795415453ea78c16f1f182c7d/A of 5326183795415453ea78c16f1f182c7d into 25fdd2bdc34c4734af9886077274bbf4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:37,311 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:37,311 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/A, priority=12, startTime=1680360876857; duration=0sec 2023-04-01 14:54:37,312 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:37,317 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#C#compaction#213 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:54:37,328 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/0d06158a6a6c4dc1aa4889cdea14f444 2023-04-01 14:54:37,337 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/b3bc17bd55f84da5af1198debd8081ce as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/b3bc17bd55f84da5af1198debd8081ce 2023-04-01 14:54:37,342 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5326183795415453ea78c16f1f182c7d/C of 5326183795415453ea78c16f1f182c7d into b3bc17bd55f84da5af1198debd8081ce(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:37,342 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:37,342 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/C, priority=12, startTime=1680360876857; duration=0sec 2023-04-01 14:54:37,342 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:37,344 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/ad54c2b2667d432e9f16ac65545951dc 2023-04-01 14:54:37,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360937507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360937508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360937510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360937579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:37,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360937579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,758 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/04cbcefa475b475ba724f7b54f396737 2023-04-01 14:54:37,763 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/0d06158a6a6c4dc1aa4889cdea14f444 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/0d06158a6a6c4dc1aa4889cdea14f444 2023-04-01 14:54:37,767 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/0d06158a6a6c4dc1aa4889cdea14f444, entries=150, sequenceid=106, filesize=11.6 K 2023-04-01 14:54:37,768 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/ad54c2b2667d432e9f16ac65545951dc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/ad54c2b2667d432e9f16ac65545951dc 2023-04-01 14:54:37,774 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/ad54c2b2667d432e9f16ac65545951dc, entries=150, sequenceid=106, filesize=11.6 K 2023-04-01 14:54:37,775 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/04cbcefa475b475ba724f7b54f396737 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/04cbcefa475b475ba724f7b54f396737 2023-04-01 14:54:37,778 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/04cbcefa475b475ba724f7b54f396737, entries=150, sequenceid=106, filesize=11.6 K 2023-04-01 14:54:37,779 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5326183795415453ea78c16f1f182c7d in 879ms, sequenceid=106, compaction requested=false 2023-04-01 14:54:37,779 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:37,779 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:37,780 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:37,781 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:37,781 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:37,782 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:37,782 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:37,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:37,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:37,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:37,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:37,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:37,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c726d7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:37,783 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:37,783 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:37,785 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:37,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:37,785 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,785 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:37,785 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:37,785 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:37,785 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:37,786 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:37,786 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:37,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:37,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:37,786 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:37,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:37,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:37,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:37,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:37,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:37,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:37,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:37,789 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:37,789 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:37,789 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:37,789 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:37,789 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:37,792 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:37,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:37,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:37,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:37,792 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:37,793 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:37,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:37,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:37,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:37,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:37,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:37,793 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:37,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:37,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:37,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:37,794 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:37,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:37,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:37,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:37,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:37,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:37,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:37,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:37,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:37,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:37,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:37,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:37,797 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:37,797 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:37,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:37,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:37,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:37,797 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:37,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:37,897 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:37,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:37,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:37,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:37,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:37,901 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:37,901 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:37,901 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:37,901 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:37,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:37,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:37,903 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:37,903 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:37,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:37,903 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:37,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:37,903 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:37,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:37,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:37,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:37,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:37,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:37,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:37,905 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:37,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:37,918 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:37,918 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:37,919 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:54:37,919 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:37,919 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:37,919 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:37,919 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:37,919 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:37,919 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:38,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:38,084 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:38,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:38,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360938096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:38,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360938098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:38,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360938199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:38,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360938201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,359 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/41df6e8cbe414fd9bad29a9f2d29aea6 2023-04-01 14:54:38,372 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/602022782c3b4e46b64535e23107acd5 2023-04-01 14:54:38,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:38,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360938402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:38,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360938405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:38,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360938514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:38,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:38,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360938517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360938518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:38,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360938704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:38,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360938708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,822 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/e2a1b28137ed424fb7cda39b73752be9 2023-04-01 14:54:38,827 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/41df6e8cbe414fd9bad29a9f2d29aea6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/41df6e8cbe414fd9bad29a9f2d29aea6 2023-04-01 14:54:38,830 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/41df6e8cbe414fd9bad29a9f2d29aea6, entries=150, sequenceid=127, filesize=11.6 K 2023-04-01 14:54:38,831 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/602022782c3b4e46b64535e23107acd5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/602022782c3b4e46b64535e23107acd5 2023-04-01 14:54:38,834 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/602022782c3b4e46b64535e23107acd5, entries=150, sequenceid=127, filesize=11.6 K 2023-04-01 14:54:38,835 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/e2a1b28137ed424fb7cda39b73752be9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e2a1b28137ed424fb7cda39b73752be9 2023-04-01 14:54:38,840 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e2a1b28137ed424fb7cda39b73752be9, entries=150, sequenceid=127, filesize=11.6 K 2023-04-01 14:54:38,841 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 5326183795415453ea78c16f1f182c7d in 923ms, sequenceid=127, compaction requested=true 2023-04-01 14:54:38,841 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:38,841 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:38,841 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:38,841 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:38,841 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,841 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:38,841 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:38,843 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:38,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:38,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4221d93c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:38,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:38,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:38,848 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:38,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:38,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:38,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:38,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:38,848 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,848 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:38,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:38,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:38,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:38,849 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:38,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:38,852 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:38,852 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:38,852 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:38,852 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:38,853 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:38,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:38,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:38,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:38,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:38,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:38,855 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:38,856 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,862 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:38,862 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:38,862 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:38,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:38,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:38,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:38,862 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:38,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:38,862 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:38,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:38,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:38,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:38,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:38,862 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:38,863 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:38,963 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:38,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:38,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:38,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:38,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:38,967 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:38,967 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:38,967 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:38,967 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:38,967 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:38,967 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:38,968 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:38,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:38,969 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:38,969 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:38,969 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:38,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:38,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:38,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:38,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:38,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:38,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:38,970 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:38,971 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:38,971 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:38,971 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:38,971 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:38,971 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:38,971 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:38,972 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:54:38,972 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:38,973 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:38,973 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:38,973 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:38,973 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:38,973 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:39,159 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 14:54:39,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:39,211 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:39,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:39,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360939265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:39,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360939265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:39,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360939370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:39,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360939370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,388 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/d527fedf69504d13b3189700daad15de 2023-04-01 14:54:39,400 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/76628ee139fa4a29818723dc619e5d2c 2023-04-01 14:54:39,410 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/825a09aeaade48f580b3ee46a4dad93c 2023-04-01 14:54:39,415 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/d527fedf69504d13b3189700daad15de as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d527fedf69504d13b3189700daad15de 2023-04-01 14:54:39,419 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d527fedf69504d13b3189700daad15de, entries=150, sequenceid=141, filesize=11.8 K 2023-04-01 14:54:39,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/76628ee139fa4a29818723dc619e5d2c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/76628ee139fa4a29818723dc619e5d2c 2023-04-01 14:54:39,423 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/76628ee139fa4a29818723dc619e5d2c, entries=150, sequenceid=141, filesize=11.8 K 2023-04-01 14:54:39,423 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/825a09aeaade48f580b3ee46a4dad93c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/825a09aeaade48f580b3ee46a4dad93c 2023-04-01 14:54:39,427 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/825a09aeaade48f580b3ee46a4dad93c, entries=150, sequenceid=141, filesize=11.8 K 2023-04-01 14:54:39,427 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5326183795415453ea78c16f1f182c7d in 455ms, sequenceid=141, compaction requested=true 2023-04-01 14:54:39,427 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:39,427 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:39,427 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:39,427 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:39,427 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:39,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:39,437 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:39,437 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:39,437 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:39,438 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:39,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:39,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26bf3aec[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:39,439 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:39,439 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:39,440 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:39,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:39,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:39,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:39,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:39,441 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,441 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:39,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:39,444 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:39,444 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:39,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:39,446 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:39,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:39,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:39,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:39,447 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:39,448 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:39,448 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:39,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:39,448 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:39,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:39,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:39,450 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:39,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:39,454 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:39,454 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,454 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:39,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:39,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:39,454 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:39,454 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:39,454 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:39,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:39,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:39,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:39,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:39,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:39,455 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:39,455 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:39,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:39,555 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:39,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:39,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:39,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:39,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:39,559 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:39,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:39,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:39,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:39,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:39,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:39,566 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:39,566 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:39,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:39,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:39,567 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,567 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:39,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:39,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:39,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:39,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:39,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:39,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:39,567 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:39,567 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:39,567 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:39,568 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:39,568 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:39,568 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:39,568 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:39,568 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:54:39,568 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:39,568 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:39,568 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:39,568 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:39,568 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:39,568 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:39,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:39,575 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:39,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:39,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:39,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360939614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360939614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:39,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360939718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:39,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360939718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:39,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360939920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:39,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360939921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:39,975 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/3ae75540f92a4a228df4f493edff6d2c 2023-04-01 14:54:39,994 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/bac363b605ab48be92a67279cbccd37b 2023-04-01 14:54:40,014 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/926775ebef074768a01f9238d3132ed1 2023-04-01 14:54:40,018 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/3ae75540f92a4a228df4f493edff6d2c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/3ae75540f92a4a228df4f493edff6d2c 2023-04-01 14:54:40,022 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/3ae75540f92a4a228df4f493edff6d2c, entries=150, sequenceid=159, filesize=11.8 K 2023-04-01 14:54:40,024 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/bac363b605ab48be92a67279cbccd37b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/bac363b605ab48be92a67279cbccd37b 2023-04-01 14:54:40,038 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/bac363b605ab48be92a67279cbccd37b, entries=150, sequenceid=159, filesize=11.8 K 2023-04-01 14:54:40,039 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/926775ebef074768a01f9238d3132ed1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/926775ebef074768a01f9238d3132ed1 2023-04-01 14:54:40,043 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/926775ebef074768a01f9238d3132ed1, entries=150, sequenceid=159, filesize=11.8 K 2023-04-01 14:54:40,044 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 5326183795415453ea78c16f1f182c7d in 476ms, sequenceid=159, compaction requested=true 2023-04-01 14:54:40,044 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:40,044 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:40,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:40,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:40,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:40,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:40,049 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:40,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:40,049 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:40,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:40,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:40,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:40,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:40,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2061bf92[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:40,051 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:40,051 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:40,052 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:40,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:40,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:40,053 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,053 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:40,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,054 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:40,054 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:40,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:40,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:40,054 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:40,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:40,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:40,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:40,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:40,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:40,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:40,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:40,056 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:40,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:40,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:40,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:40,056 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:40,059 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:40,059 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:40,059 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:40,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:40,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:40,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:40,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:40,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:40,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:40,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:40,062 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:40,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:40,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:40,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:40,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:40,064 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:40,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:40,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:40,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,066 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:40,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:40,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:40,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:40,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:40,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:40,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:40,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:40,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:40,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:40,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:40,066 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:40,066 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:40,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:40,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:40,167 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:40,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:40,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:40,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:40,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:40,170 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:40,170 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:40,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:40,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:40,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:40,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:40,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:40,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:40,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:40,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:40,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:40,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,174 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:40,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:40,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:40,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:40,175 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:40,175 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:40,175 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:40,175 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:40,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:40,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:40,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:40,176 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:54:40,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:40,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:40,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:40,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:40,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:40,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:40,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:40,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:40,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:40,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360940264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:40,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360940265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:40,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360940370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:40,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360940370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:40,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360940524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:40,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360940524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,527 DEBUG [Thread-1542] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:40,528 DEBUG [Thread-1536] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:40,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:40,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360940531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,535 DEBUG [Thread-1538] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:40,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:40,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360940573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:40,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360940573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,598 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/f300dddd81bc4baa81ee3b7c60275d1f 2023-04-01 14:54:40,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:40,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360940877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:40,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:40,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360940878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,009 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/806f135bd38f4f4ab828f145db7b7407 2023-04-01 14:54:41,031 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/5763084e80664913b007203107214c95 2023-04-01 14:54:41,036 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/f300dddd81bc4baa81ee3b7c60275d1f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/f300dddd81bc4baa81ee3b7c60275d1f 2023-04-01 14:54:41,040 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/f300dddd81bc4baa81ee3b7c60275d1f, entries=150, sequenceid=173, filesize=11.8 K 2023-04-01 14:54:41,041 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/806f135bd38f4f4ab828f145db7b7407 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/806f135bd38f4f4ab828f145db7b7407 2023-04-01 14:54:41,045 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/806f135bd38f4f4ab828f145db7b7407, entries=150, sequenceid=173, filesize=11.8 K 2023-04-01 14:54:41,046 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/5763084e80664913b007203107214c95 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/5763084e80664913b007203107214c95 2023-04-01 14:54:41,050 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/5763084e80664913b007203107214c95, entries=150, sequenceid=173, filesize=11.8 K 2023-04-01 14:54:41,051 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5326183795415453ea78c16f1f182c7d in 875ms, sequenceid=173, compaction requested=true 2023-04-01 14:54:41,051 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:41,051 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:41,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:41,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:41,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:41,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:41,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,053 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:41,053 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:41,053 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:41,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:41,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a5ac4d7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:41,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:41,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:41,057 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:41,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:41,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:41,057 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:41,057 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:41,057 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:41,057 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,057 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:41,058 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:41,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:41,061 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:41,061 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:41,061 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:41,061 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:41,061 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:41,063 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:41,063 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:41,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:41,063 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:41,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:41,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:41,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:41,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:41,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:41,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:41,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:41,064 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:41,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:41,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:41,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:41,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:41,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:41,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:41,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:41,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:41,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:41,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:41,071 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:41,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:41,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:41,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:41,071 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:41,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:41,072 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:41,072 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:41,072 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:41,172 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:41,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:41,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:41,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:41,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:41,175 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:41,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:41,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:41,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:41,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:41,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:41,177 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:41,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:41,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:41,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:41,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:41,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:41,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:41,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:41,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:41,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:41,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:41,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:41,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:41,179 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:41,179 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:41,179 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:54:41,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:41,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:41,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:41,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:41,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:41,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:41,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:41,389 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:41,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:41,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360941412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:41,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360941415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:41,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360941516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:41,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360941520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,594 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/a67a308f5d9344bbb99f07bafe7d8b50 2023-04-01 14:54:41,632 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/3cedd6a2ff8c4527b9c48b4ddc185b95 2023-04-01 14:54:41,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:41,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360941720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:41,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:41,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360941723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:42,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360942024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:42,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360942029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,058 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/2f28591410814215b2c28f01e5edfbbe 2023-04-01 14:54:42,063 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/a67a308f5d9344bbb99f07bafe7d8b50 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/a67a308f5d9344bbb99f07bafe7d8b50 2023-04-01 14:54:42,066 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/a67a308f5d9344bbb99f07bafe7d8b50, entries=150, sequenceid=191, filesize=11.8 K 2023-04-01 14:54:42,067 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/3cedd6a2ff8c4527b9c48b4ddc185b95 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cedd6a2ff8c4527b9c48b4ddc185b95 2023-04-01 14:54:42,071 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cedd6a2ff8c4527b9c48b4ddc185b95, entries=150, sequenceid=191, filesize=11.8 K 2023-04-01 14:54:42,072 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/2f28591410814215b2c28f01e5edfbbe as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/2f28591410814215b2c28f01e5edfbbe 2023-04-01 14:54:42,074 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/2f28591410814215b2c28f01e5edfbbe, entries=150, sequenceid=191, filesize=11.8 K 2023-04-01 14:54:42,075 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 5326183795415453ea78c16f1f182c7d in 896ms, sequenceid=191, compaction requested=true 2023-04-01 14:54:42,075 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:42,075 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:42,075 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:42,075 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:42,075 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,075 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:42,075 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:42,077 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,077 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:42,077 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:42,077 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:42,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:42,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@267e1899[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:42,079 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:42,079 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:42,080 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:42,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:42,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:42,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:42,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:42,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:42,081 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,081 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:42,082 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,082 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:42,082 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:42,082 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:42,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:42,084 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:42,084 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:42,084 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:42,084 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:42,084 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:42,085 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:42,085 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:42,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:42,085 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:42,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:42,086 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:42,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:42,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:42,087 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:42,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:42,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:42,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:42,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:42,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:42,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:42,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:42,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:42,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:42,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:42,089 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:42,090 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:42,090 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:42,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:42,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:42,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:42,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:42,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:42,190 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:42,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:42,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:42,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:42,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:42,193 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:42,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:42,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:42,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:42,194 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:42,194 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:42,196 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:42,196 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:42,196 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,197 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:42,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:42,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:42,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:42,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:42,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:42,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:42,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:42,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:42,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:42,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:42,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:42,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:42,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:42,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:42,198 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:54:42,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:42,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:42,199 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:42,199 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:42,199 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:42,199 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:42,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:42,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:42,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:42,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360942571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:42,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360942573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,614 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/95e74d6a51e64c87a9d5bc5836cc7428 2023-04-01 14:54:42,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:42,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360942678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:42,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360942678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:42,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60138 deadline: 1680360942883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:42,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:42,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360942884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,024 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/f17fa2a3cb424b5083aa7637e4f549a9 2023-04-01 14:54:43,035 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/fde4113321c44c2797ee94573a67c79a 2023-04-01 14:54:43,039 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/95e74d6a51e64c87a9d5bc5836cc7428 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95e74d6a51e64c87a9d5bc5836cc7428 2023-04-01 14:54:43,043 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95e74d6a51e64c87a9d5bc5836cc7428, entries=150, sequenceid=205, filesize=11.8 K 2023-04-01 14:54:43,043 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/f17fa2a3cb424b5083aa7637e4f549a9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/f17fa2a3cb424b5083aa7637e4f549a9 2023-04-01 14:54:43,047 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/f17fa2a3cb424b5083aa7637e4f549a9, entries=150, sequenceid=205, filesize=11.8 K 2023-04-01 14:54:43,048 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/fde4113321c44c2797ee94573a67c79a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fde4113321c44c2797ee94573a67c79a 2023-04-01 14:54:43,051 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fde4113321c44c2797ee94573a67c79a, entries=150, sequenceid=205, filesize=11.8 K 2023-04-01 14:54:43,052 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5326183795415453ea78c16f1f182c7d in 854ms, sequenceid=205, compaction requested=true 2023-04-01 14:54:43,052 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:43,052 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:43,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:43,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:43,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,053 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:43,053 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:43,054 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:43,054 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,054 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:43,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:43,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79c93ee0[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:43,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:43,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,057 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,058 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:43,058 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:43,058 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:43,058 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,058 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:43,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:43,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:43,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:43,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:43,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:43,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:43,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:43,062 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:43,062 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:43,063 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:43,063 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:43,063 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:43,064 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:43,064 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,064 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:43,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:43,064 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:43,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:43,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:43,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,066 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:43,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:43,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:43,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:43,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:43,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:43,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,068 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:43,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:43,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:43,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:43,068 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:43,069 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:43,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:43,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:43,169 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:43,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:43,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:43,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:43,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:43,172 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:43,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:43,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:43,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:43,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,173 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:43,175 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:43,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:43,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:43,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:43,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:43,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:43,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:43,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:43,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:43,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:43,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:43,177 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:43,177 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:43,177 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:54:43,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:43,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:43,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:43,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:43,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:43,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:43,187 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/d4c8b111a6324586b8743cccfcccb648 2023-04-01 14:54:43,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:43,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:43,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:43,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360943211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:43,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360943217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:43,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360943318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:43,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360943322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:43,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60128 deadline: 1680360943522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:43,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360943526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,597 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/3dbb8533546a4063b01fe0006a55d462 2023-04-01 14:54:43,607 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/c9848e121bbc4dee981131070de1f14f 2023-04-01 14:54:43,612 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/d4c8b111a6324586b8743cccfcccb648 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d4c8b111a6324586b8743cccfcccb648 2023-04-01 14:54:43,615 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d4c8b111a6324586b8743cccfcccb648, entries=150, sequenceid=223, filesize=11.8 K 2023-04-01 14:54:43,616 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/3dbb8533546a4063b01fe0006a55d462 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3dbb8533546a4063b01fe0006a55d462 2023-04-01 14:54:43,619 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3dbb8533546a4063b01fe0006a55d462, entries=150, sequenceid=223, filesize=11.8 K 2023-04-01 14:54:43,620 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/c9848e121bbc4dee981131070de1f14f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/c9848e121bbc4dee981131070de1f14f 2023-04-01 14:54:43,623 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/c9848e121bbc4dee981131070de1f14f, entries=150, sequenceid=223, filesize=11.8 K 2023-04-01 14:54:43,624 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 5326183795415453ea78c16f1f182c7d in 447ms, sequenceid=223, compaction requested=true 2023-04-01 14:54:43,625 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:43,625 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:43,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:43,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:43,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:43,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:43,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,626 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:43,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:43,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:43,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54f98364[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:43,629 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:43,629 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,630 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:43,630 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,630 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:43,631 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,631 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:43,631 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:43,631 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:43,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:43,635 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:43,635 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:43,635 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:43,635 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:43,635 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:43,637 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,637 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:43,637 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:43,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:43,638 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,639 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:43,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:43,641 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:43,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:43,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:43,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:43,642 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:43,642 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:43,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:43,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:43,742 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:43,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:43,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:43,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:43,746 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:43,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:43,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:43,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:43,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:43,750 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:43,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:43,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:43,751 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,751 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:43,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:43,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:43,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:43,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:43,752 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:43,752 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:43,752 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:54:43,752 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:43,753 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:43,753 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:43,753 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:43,753 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:43,753 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:43,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:43,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:43,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:43,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360943909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:43,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:43,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360943910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:44,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360944015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:44,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360944015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,161 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/9c5b3bb9fa8344799e41dde0e7c51255 2023-04-01 14:54:44,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:44,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360944220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:44,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360944220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:44,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360944522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:44,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360944523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:44,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360944529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,533 DEBUG [Thread-1536] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:44,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:44,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360944557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,561 DEBUG [Thread-1538] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:44,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:44,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360944564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,568 DEBUG [Thread-1542] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:44,572 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/94c0ea64281346729d4d02403741ec6f 2023-04-01 14:54:44,602 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/a06599a032cb49888f837b3ae18cc466 2023-04-01 14:54:44,609 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/9c5b3bb9fa8344799e41dde0e7c51255 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/9c5b3bb9fa8344799e41dde0e7c51255 2023-04-01 14:54:44,614 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/9c5b3bb9fa8344799e41dde0e7c51255, entries=100, sequenceid=237, filesize=9.4 K 2023-04-01 14:54:44,615 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/94c0ea64281346729d4d02403741ec6f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/94c0ea64281346729d4d02403741ec6f 2023-04-01 14:54:44,621 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/94c0ea64281346729d4d02403741ec6f, entries=100, sequenceid=237, filesize=9.4 K 2023-04-01 14:54:44,622 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/a06599a032cb49888f837b3ae18cc466 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a06599a032cb49888f837b3ae18cc466 2023-04-01 14:54:44,626 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a06599a032cb49888f837b3ae18cc466, entries=100, sequenceid=237, filesize=9.4 K 2023-04-01 14:54:44,627 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5326183795415453ea78c16f1f182c7d in 874ms, sequenceid=237, compaction requested=true 2023-04-01 14:54:44,627 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:44,627 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:44,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:44,630 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:44,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:44,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:44,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:44,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68399950[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:44,632 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:44,632 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:44,633 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:44,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:44,633 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:44,633 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:44,633 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:44,633 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,634 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:44,634 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:44,634 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:44,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,634 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:44,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:44,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:44,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:44,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:44,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:44,637 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:44,638 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:44,638 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:44,638 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:44,638 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:44,639 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:44,639 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:44,639 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:44,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:44,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:44,640 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:44,641 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,645 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,645 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:44,645 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:44,645 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,645 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:44,645 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:44,645 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:44,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:44,645 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:44,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:44,645 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:44,646 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:44,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:44,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:44,646 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:44,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:44,646 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:44,746 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:44,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:44,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:44,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:44,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:44,750 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:44,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:44,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:44,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:44,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:44,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:44,751 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:44,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:44,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:44,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:44,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:44,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:44,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:44,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:44,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:44,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:44,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:44,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:44,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:44,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:44,753 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:44,754 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:44,754 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:54:44,754 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:44,755 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:44,755 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:44,755 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:44,755 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:44,755 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:45,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:45,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:45,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:45,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360945065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:45,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360945070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,162 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/79a5be2ce6734d67bda232a14038761b 2023-04-01 14:54:45,171 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/8de146800bb449b78dcae7824c2b9190 2023-04-01 14:54:45,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360945171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:45,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360945175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:45,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360945375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:45,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360945380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,581 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/a959a00641284f90b9024d10fbafd7dd 2023-04-01 14:54:45,585 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/79a5be2ce6734d67bda232a14038761b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/79a5be2ce6734d67bda232a14038761b 2023-04-01 14:54:45,590 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/79a5be2ce6734d67bda232a14038761b, entries=150, sequenceid=255, filesize=11.8 K 2023-04-01 14:54:45,591 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/8de146800bb449b78dcae7824c2b9190 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/8de146800bb449b78dcae7824c2b9190 2023-04-01 14:54:45,594 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/8de146800bb449b78dcae7824c2b9190, entries=150, sequenceid=255, filesize=11.8 K 2023-04-01 14:54:45,595 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/a959a00641284f90b9024d10fbafd7dd as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a959a00641284f90b9024d10fbafd7dd 2023-04-01 14:54:45,598 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a959a00641284f90b9024d10fbafd7dd, entries=150, sequenceid=255, filesize=11.8 K 2023-04-01 14:54:45,599 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 5326183795415453ea78c16f1f182c7d in 845ms, sequenceid=255, compaction requested=true 2023-04-01 14:54:45,599 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:45,599 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:45,601 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,601 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,601 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,601 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:45,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:45,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:45,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ed5fe1e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:45,603 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:45,603 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,604 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:45,604 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:45,604 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:45,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:45,606 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:45,606 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,606 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,606 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:45,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:45,609 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:45,609 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:45,609 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:45,609 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:45,609 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:45,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:45,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:45,612 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,613 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:45,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:45,615 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:45,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:45,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:45,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:45,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:45,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:45,616 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:45,616 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:45,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:45,682 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 14:54:45,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:45,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:45,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:45,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:45,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:45,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:45,716 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:45,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:45,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:45,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:45,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:45,719 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:45,720 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:45,720 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:45,720 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:45,720 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,720 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,721 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:45,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:45,721 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,721 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:45,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:45,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:45,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:45,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:45,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:45,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:45,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:45,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:45,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:45,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:45,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:45,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:45,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:45,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:45,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:45,725 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:45,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:45,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:45,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360945719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,727 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:45,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@edbf91a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:45,727 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,728 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:45,728 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:45,728 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,728 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:45,728 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:45,730 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,730 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:45,730 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,730 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:45,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:45,732 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:45,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:45,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:45,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:45,732 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:45,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:45,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:45,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360945727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:45,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:45,734 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:45,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:45,735 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:45,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:45,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:45,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:45,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:45,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:45,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:45,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:45,739 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:45,739 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:45,739 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,739 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,739 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:45,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:45,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360945828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,839 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:45,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:45,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360945834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:45,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:45,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:45,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:45,845 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:45,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:45,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:45,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:45,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,847 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:45,847 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:45,847 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,847 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:45,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:45,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:45,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:45,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:45,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:45,850 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:45,851 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,851 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:45,852 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,852 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:45,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:45,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1621b906[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:45,853 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:45,853 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,858 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,859 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:45,859 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:45,859 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:45,859 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,859 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:45,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:45,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:45,862 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:45,862 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:45,862 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:45,862 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:45,862 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:45,864 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,864 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:45,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,864 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,864 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,865 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:45,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:45,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:45,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:45,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:45,868 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:45,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,868 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:45,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:45,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:45,868 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:45,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,968 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:45,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:45,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:45,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:45,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:45,980 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:45,980 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:45,980 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:45,980 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:45,981 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,981 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:45,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:45,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:45,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:45,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:45,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:45,985 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:45,985 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:45,985 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:45,986 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:45,986 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:45,986 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:45,987 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:45,987 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:45,987 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:45,987 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:45,987 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:45,987 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,987 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:45,987 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:45,988 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,989 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,989 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,989 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:45,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:45,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3276518b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:45,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:45,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,992 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:45,992 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:45,992 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,992 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:45,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:45,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:45,993 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:45,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:45,996 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:45,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:45,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:45,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:45,996 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:45,999 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:45,999 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,000 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,000 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,002 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,005 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,005 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:46,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:46,005 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,005 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,005 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,005 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,005 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,005 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,005 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,005 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,005 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:46,005 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:46,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:46,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360946035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:46,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360946040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/69c790d897f94ae9be33e86ab9c115d4 2023-04-01 14:54:46,105 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:46,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:46,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:46,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:46,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:46,109 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:46,109 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,109 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:46,109 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:46,109 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,109 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,110 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,110 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,110 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,110 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:46,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:46,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:46,112 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,112 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:46,112 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:46,112 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:46,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:46,114 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,114 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,114 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,114 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23c26cd5[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:46,115 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:46,115 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,117 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,117 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:46,118 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:46,118 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:46,118 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,118 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:46,119 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:46,119 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,119 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,119 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:46,121 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:46,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:46,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:46,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:46,122 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:46,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,124 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,126 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,129 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,129 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,129 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,129 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:46,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:46,129 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,129 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,129 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:46,129 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:46,129 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,129 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,130 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,130 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,230 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:46,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:46,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:46,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:46,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:46,233 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:46,233 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,233 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:46,233 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:46,233 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,233 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,235 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,235 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,236 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,236 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:46,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:46,237 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,237 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:46,237 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:46,237 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:46,239 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,239 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,239 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,239 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:46,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7200529e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:46,241 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:46,241 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,242 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,242 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:46,242 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:46,242 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,242 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:46,242 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:46,247 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,247 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,247 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,247 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:46,249 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:46,249 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:46,249 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:46,249 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:46,249 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:46,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,251 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,254 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,254 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,255 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,255 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,255 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,255 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,255 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,255 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,255 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,255 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:46,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:46,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,255 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:46,255 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:46,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:46,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:46,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360946342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:46,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360946344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,355 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:46,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:46,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:46,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:46,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:46,358 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:46,358 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,359 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:46,359 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:46,359 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,359 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,361 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,361 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:46,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:46,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:46,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:46,362 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,362 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:46,362 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:46,362 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,363 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:46,363 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,366 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,366 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:46,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f22c51[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:46,367 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:46,367 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,368 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,368 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,368 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:46,368 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:46,368 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:46,368 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:46,370 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,370 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,370 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,370 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:46,372 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:46,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:46,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:46,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:46,373 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:46,374 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,374 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,374 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,375 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,376 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,379 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:46,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:46,380 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,380 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,380 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,380 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,380 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:46,380 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:46,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,480 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:46,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:46,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:46,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:46,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:46,483 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:46,483 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,483 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:46,483 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:46,484 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,484 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,485 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,485 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,485 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,486 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:46,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:46,487 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,487 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:46,487 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:46,487 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:46,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:46,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6908c2ed[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:46,490 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:46,491 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,492 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:46,492 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,492 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:46,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:46,498 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:46,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:46,500 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:46,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:46,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:46,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:46,501 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:46,502 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/1a16b22ed4ba4e9986d99404b39d9bac 2023-04-01 14:54:46,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,503 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,504 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,507 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:46,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:46,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,507 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:46,508 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:46,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:46,608 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:46,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:46,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:46,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:46,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:46,611 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:46,611 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,611 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:46,611 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:46,611 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,611 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,613 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,613 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,613 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,613 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:46,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:46,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:46,614 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:46,614 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,614 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:46,614 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:46,614 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:46,614 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,614 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:46,615 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:46,615 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:46,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:46,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:46,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:46,617 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:46,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@599cbfdd[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:46,619 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:46,619 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,620 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,620 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:46,620 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:46,620 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:46,620 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:46,621 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:46,622 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:46,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:46,624 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:46,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:46,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:46,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:46,624 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:46,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,626 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,627 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:46,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:46,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,643 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,643 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:46,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:46,644 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:46,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,744 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:46,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:46,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:46,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:46,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:46,760 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:46,760 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,760 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:46,760 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:46,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,762 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,763 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:46,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:46,786 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,787 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:46,787 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:46,787 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:46,789 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:46,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29f892d2[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:46,793 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:46,793 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,794 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,794 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:46,794 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:46,794 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:46,795 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,795 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:46,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,797 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:46,797 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,797 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:46,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:46,802 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:46,802 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:46,802 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:46,802 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:46,802 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:46,803 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,803 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,803 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,804 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,805 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:46,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:46,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,808 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:46,808 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:46,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,809 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,809 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,809 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,809 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,809 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:46,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360946847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:46,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360946849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,909 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:46,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:46,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:46,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:46,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:46,912 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:46,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:46,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:46,913 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,913 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,914 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,914 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,914 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/f9b5982cae604683809b82fc4f34f0b5 2023-04-01 14:54:46,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:46,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:46,916 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:46,916 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:46,916 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:46,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:46,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:46,918 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,918 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:46,918 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:46,918 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:46,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:46,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:46,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,921 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:46,921 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/69c790d897f94ae9be33e86ab9c115d4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/69c790d897f94ae9be33e86ab9c115d4 2023-04-01 14:54:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:46,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5110a318[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:46,925 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:46,925 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:46,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:46,928 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,929 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:46,929 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,929 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,932 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/69c790d897f94ae9be33e86ab9c115d4, entries=200, sequenceid=270, filesize=14.3 K 2023-04-01 14:54:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:46,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:46,933 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:46,933 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:46,933 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:46,933 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:46,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/1a16b22ed4ba4e9986d99404b39d9bac as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1a16b22ed4ba4e9986d99404b39d9bac 2023-04-01 14:54:46,933 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:46,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:46,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,936 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,937 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:46,940 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:46,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:46,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:46,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:46,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:46,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:46,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:46,940 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:46,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:46,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:46,941 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:46,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1a16b22ed4ba4e9986d99404b39d9bac, entries=150, sequenceid=270, filesize=11.9 K 2023-04-01 14:54:46,941 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:46,941 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:46,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/f9b5982cae604683809b82fc4f34f0b5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f9b5982cae604683809b82fc4f34f0b5 2023-04-01 14:54:46,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f9b5982cae604683809b82fc4f34f0b5, entries=150, sequenceid=270, filesize=11.9 K 2023-04-01 14:54:46,946 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 5326183795415453ea78c16f1f182c7d in 1265ms, sequenceid=270, compaction requested=true 2023-04-01 14:54:46,946 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:46,946 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:46,946 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 14:54:46,946 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-01 14:54:46,946 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 12 store files, 0 compacting, 12 eligible, 16 blocking 2023-04-01 14:54:46,946 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 12 store files, 0 compacting, 12 eligible, 16 blocking 2023-04-01 14:54:46,950 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 117654 starting at candidate #0 after considering 52 permutations with 52 in ratio 2023-04-01 14:54:46,950 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/A is initiating minor compaction 2023-04-01 14:54:46,951 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/A in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,951 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/25fdd2bdc34c4734af9886077274bbf4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/0d06158a6a6c4dc1aa4889cdea14f444, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/41df6e8cbe414fd9bad29a9f2d29aea6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d527fedf69504d13b3189700daad15de, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/3ae75540f92a4a228df4f493edff6d2c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/f300dddd81bc4baa81ee3b7c60275d1f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/a67a308f5d9344bbb99f07bafe7d8b50, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95e74d6a51e64c87a9d5bc5836cc7428, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d4c8b111a6324586b8743cccfcccb648, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/9c5b3bb9fa8344799e41dde0e7c51255] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=114.9 K 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 25fdd2bdc34c4734af9886077274bbf4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 0d06158a6a6c4dc1aa4889cdea14f444, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 41df6e8cbe414fd9bad29a9f2d29aea6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d527fedf69504d13b3189700daad15de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 3ae75540f92a4a228df4f493edff6d2c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting f300dddd81bc4baa81ee3b7c60275d1f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting a67a308f5d9344bbb99f07bafe7d8b50, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 95e74d6a51e64c87a9d5bc5836cc7428, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d4c8b111a6324586b8743cccfcccb648, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 9c5b3bb9fa8344799e41dde0e7c51255, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=237 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 117654 starting at candidate #0 after considering 52 permutations with 52 in ratio 2023-04-01 14:54:46,951 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/C is initiating minor compaction 2023-04-01 14:54:46,951 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/C in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,952 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/b3bc17bd55f84da5af1198debd8081ce, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/04cbcefa475b475ba724f7b54f396737, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e2a1b28137ed424fb7cda39b73752be9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/825a09aeaade48f580b3ee46a4dad93c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/926775ebef074768a01f9238d3132ed1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/5763084e80664913b007203107214c95, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/2f28591410814215b2c28f01e5edfbbe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fde4113321c44c2797ee94573a67c79a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/c9848e121bbc4dee981131070de1f14f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a06599a032cb49888f837b3ae18cc466] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=114.9 K 2023-04-01 14:54:46,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b3bc17bd55f84da5af1198debd8081ce, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92 2023-04-01 14:54:46,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 04cbcefa475b475ba724f7b54f396737, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106 2023-04-01 14:54:46,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e2a1b28137ed424fb7cda39b73752be9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127 2023-04-01 14:54:46,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 825a09aeaade48f580b3ee46a4dad93c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141 2023-04-01 14:54:46,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 926775ebef074768a01f9238d3132ed1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159 2023-04-01 14:54:46,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 5763084e80664913b007203107214c95, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173 2023-04-01 14:54:46,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2f28591410814215b2c28f01e5edfbbe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191 2023-04-01 14:54:46,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting fde4113321c44c2797ee94573a67c79a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205 2023-04-01 14:54:46,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c9848e121bbc4dee981131070de1f14f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223 2023-04-01 14:54:46,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a06599a032cb49888f837b3ae18cc466, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=237 2023-04-01 14:54:46,970 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#C#compaction#246 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:46,973 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#A#compaction#247 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:46,985 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/e99527f73a894d42a124a915a7a3a406 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e99527f73a894d42a124a915a7a3a406 2023-04-01 14:54:46,989 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 5326183795415453ea78c16f1f182c7d/C of 5326183795415453ea78c16f1f182c7d into e99527f73a894d42a124a915a7a3a406(size=12.2 K), total size for store is 35.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:46,989 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:46,989 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/C, priority=4, startTime=1680360886946; duration=0sec 2023-04-01 14:54:46,989 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:46,989 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 12 store files, 0 compacting, 12 eligible, 16 blocking 2023-04-01 14:54:46,992 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 117654 starting at candidate #0 after considering 52 permutations with 52 in ratio 2023-04-01 14:54:46,992 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/B is initiating minor compaction 2023-04-01 14:54:46,993 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/B in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:46,993 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/4b6e5e79fe3f45c18249d843b2118a48, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/ad54c2b2667d432e9f16ac65545951dc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/602022782c3b4e46b64535e23107acd5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/76628ee139fa4a29818723dc619e5d2c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/bac363b605ab48be92a67279cbccd37b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/806f135bd38f4f4ab828f145db7b7407, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cedd6a2ff8c4527b9c48b4ddc185b95, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/f17fa2a3cb424b5083aa7637e4f549a9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3dbb8533546a4063b01fe0006a55d462, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/94c0ea64281346729d4d02403741ec6f] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=114.9 K 2023-04-01 14:54:46,993 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 4b6e5e79fe3f45c18249d843b2118a48, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92 2023-04-01 14:54:46,993 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ad54c2b2667d432e9f16ac65545951dc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106 2023-04-01 14:54:46,993 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 602022782c3b4e46b64535e23107acd5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127 2023-04-01 14:54:46,993 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 76628ee139fa4a29818723dc619e5d2c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141 2023-04-01 14:54:46,993 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting bac363b605ab48be92a67279cbccd37b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159 2023-04-01 14:54:46,993 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 806f135bd38f4f4ab828f145db7b7407, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173 2023-04-01 14:54:46,993 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3cedd6a2ff8c4527b9c48b4ddc185b95, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191 2023-04-01 14:54:46,993 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f17fa2a3cb424b5083aa7637e4f549a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205 2023-04-01 14:54:46,994 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3dbb8533546a4063b01fe0006a55d462, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223 2023-04-01 14:54:46,994 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 94c0ea64281346729d4d02403741ec6f, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=237 2023-04-01 14:54:46,994 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/1d5b2eef22034e96bd762f9104ec25bb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/1d5b2eef22034e96bd762f9104ec25bb 2023-04-01 14:54:47,002 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 5326183795415453ea78c16f1f182c7d/A of 5326183795415453ea78c16f1f182c7d into 1d5b2eef22034e96bd762f9104ec25bb(size=12.2 K), total size for store is 38.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:47,002 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:47,002 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/A, priority=4, startTime=1680360886946; duration=0sec 2023-04-01 14:54:47,002 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:47,014 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#B#compaction#248 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:47,041 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:47,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:47,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:47,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:47,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:47,052 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:47,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:47,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:47,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:47,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:47,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:47,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:47,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:47,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:47,055 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:47,056 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:47,056 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 14:54:47,056 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:47,056 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:47,056 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:47,056 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:47,056 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:47,057 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:47,070 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/e6773508d0d44712af6555cb59476424 2023-04-01 14:54:47,085 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/063813620c0c463b8928ada40fe87294 2023-04-01 14:54:47,426 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/cc666f9ad04448beb331868bd6933dab as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/cc666f9ad04448beb331868bd6933dab 2023-04-01 14:54:47,430 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 5326183795415453ea78c16f1f182c7d/B of 5326183795415453ea78c16f1f182c7d into cc666f9ad04448beb331868bd6933dab(size=12.2 K), total size for store is 35.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:47,431 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:47,431 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/B, priority=4, startTime=1680360886946; duration=0sec 2023-04-01 14:54:47,431 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:47,502 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/4375f41e92c045cfbb273c9db766b1d9 2023-04-01 14:54:47,507 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/e6773508d0d44712af6555cb59476424 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e6773508d0d44712af6555cb59476424 2023-04-01 14:54:47,511 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e6773508d0d44712af6555cb59476424, entries=150, sequenceid=289, filesize=11.9 K 2023-04-01 14:54:47,512 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/063813620c0c463b8928ada40fe87294 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/063813620c0c463b8928ada40fe87294 2023-04-01 14:54:47,517 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/063813620c0c463b8928ada40fe87294, entries=150, sequenceid=289, filesize=11.9 K 2023-04-01 14:54:47,518 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/4375f41e92c045cfbb273c9db766b1d9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4375f41e92c045cfbb273c9db766b1d9 2023-04-01 14:54:47,521 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4375f41e92c045cfbb273c9db766b1d9, entries=150, sequenceid=289, filesize=11.9 K 2023-04-01 14:54:47,522 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 5326183795415453ea78c16f1f182c7d in 466ms, sequenceid=289, compaction requested=true 2023-04-01 14:54:47,522 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:47,522 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:47,522 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:47,522 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:47,522 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,522 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:47,522 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:47,524 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ea01c7d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:47,527 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:47,527 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,528 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:47,528 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:47,528 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,528 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:47,530 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:47,533 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:47,533 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:47,533 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:47,533 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:47,533 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:47,534 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,534 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,534 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,535 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,537 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:47,542 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:47,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:47,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:47,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:47,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:47,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:47,542 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:47,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:47,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,542 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:47,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,543 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,643 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:47,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:47,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:47,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:47,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:47,646 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:47,646 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:47,646 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:47,646 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:47,646 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,646 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,653 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,653 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:47,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:47,653 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,653 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:47,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:47,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:47,654 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:47,654 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:47,654 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:47,655 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:47,655 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:47,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:47,656 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,657 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42fe3623[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:47,658 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:47,658 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,659 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,659 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:47,659 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:47,659 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:47,659 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,659 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:47,661 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:47,661 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:47,661 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,661 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:47,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:47,664 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:47,664 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:47,664 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:47,664 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:47,664 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:47,666 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,666 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,666 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:47,667 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,668 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:47,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:47,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:47,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:47,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,671 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,671 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:47,671 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,671 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,671 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,670 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:47,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:47,671 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:47,671 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:47,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:47,771 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:47,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:47,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:47,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:47,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:47,774 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:47,774 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:47,774 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:47,774 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:47,775 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,775 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:47,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:47,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:47,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:47,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:47,778 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:47,778 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:47,778 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:47,778 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:47,782 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f7d42ac[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:47,782 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,788 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:47,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:47,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:47,788 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,789 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:47,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:47,790 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:47,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:47,792 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:47,793 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:47,793 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:47,793 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:47,793 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:47,794 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,794 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:47,794 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,795 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,796 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:47,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:47,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:47,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:47,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:47,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:47,798 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:47,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,798 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:47,799 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:47,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:47,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:47,862 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:54:47,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:47,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:47,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:47,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:47,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:47,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:47,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360887887, max timestamp=1680360887893Num uniques 859; 2023-04-01 14:54:47,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360887887, max timestamp=1680360887893Num uniques 859; 2023-04-01 14:54:47,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:47,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:47,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360887887, max timestamp=1680360887893Num uniques 859; 2023-04-01 14:54:47,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:47,899 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:47,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:47,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:47,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:47,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:47,903 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:47,903 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:47,903 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:47,903 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:47,904 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,904 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,906 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,906 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:47,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:47,906 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,906 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:47,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:47,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:47,908 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:47,908 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:47,908 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:47,909 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:47,909 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:47,909 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:47,909 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:47,911 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,911 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:47,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ba12c38[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:47,913 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:47,914 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,915 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,915 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:47,915 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:47,915 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:47,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:47,917 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:47,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:47,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:47,921 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:47,921 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:47,921 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:47,921 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:47,921 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:47,924 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,924 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,924 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:47,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:47,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:47,925 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,927 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:47,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:47,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,931 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:47,931 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:47,931 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:47,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:47,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:47,931 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:47,931 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:47,931 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,931 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:47,932 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:47,932 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,932 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,932 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:47,932 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:47,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:47,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:47,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680360887916, max timestamp=1680360887931Num uniques 1018; 2023-04-01 14:54:47,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 14:54:47,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680360887916, max timestamp=1680360887931Num uniques 1018; 2023-04-01 14:54:47,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 14:54:47,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680360887916, max timestamp=1680360887931Num uniques 1018; 2023-04-01 14:54:47,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-01 14:54:47,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:47,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360947966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:47,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:47,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360947967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,031 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:48,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:48,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:48,034 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:48,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:48,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:48,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:48,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:48,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:48,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:48,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:48,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:48,040 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,041 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:48,041 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:48,041 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,045 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:48,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51c84c5[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:48,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:48,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,048 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:48,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:48,049 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,049 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:48,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:48,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:48,052 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:48,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:48,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:48,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:48,052 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:48,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,058 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,058 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,058 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,058 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:48,058 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:48,058 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:48,058 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,058 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,058 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,058 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:48,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:48,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:48,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:48,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:48,058 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:48,059 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:48,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:48,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360948069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:48,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360948070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,159 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:48,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:48,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:48,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:48,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:48,163 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:48,163 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,163 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:48,163 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:48,163 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,163 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,164 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:48,165 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:48,165 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,165 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:48,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:48,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:48,167 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,167 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:48,167 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:48,167 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:48,168 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:48,169 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,169 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,169 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,169 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:48,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3660e56b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:48,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:48,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,172 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:48,172 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:48,172 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:48,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:48,174 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:48,174 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:48,174 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,174 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:48,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:48,177 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:48,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:48,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:48,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:48,177 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:48,178 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,178 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,179 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:48,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,179 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,180 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,182 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:48,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:48,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:48,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:48,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:48,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:48,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:48,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,183 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,183 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,183 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:48,183 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:48,183 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:48,183 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,183 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/e4edf08d52cf4286a8d380c797fbb989 2023-04-01 14:54:48,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:48,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360948271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:48,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360948272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/e0a13db1df324174a1e3c424dc24d498 2023-04-01 14:54:48,283 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:48,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:48,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:48,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:48,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:48,287 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:48,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:48,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:48,288 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,288 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,289 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:48,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:48,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:48,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:48,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:48,294 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,294 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:48,294 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:48,294 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,295 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,296 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,296 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73cda929[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:48,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:48,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,299 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,299 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:48,299 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:48,299 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:48,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:48,301 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:48,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:48,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:48,303 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:48,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:48,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:48,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:48,303 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:48,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:48,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:48,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:48,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:48,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:48,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:48,310 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:48,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:48,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:48,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:48,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,311 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:48,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,311 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:48,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,411 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:48,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:48,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:48,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:48,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:48,415 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:48,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:48,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:48,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:48,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:48,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,418 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:48,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:48,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:48,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:48,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:48,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:48,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:48,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:48,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:48,421 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,422 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,422 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:48,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff9b84[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:48,423 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:48,423 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,426 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:48,426 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,426 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:48,427 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,427 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:48,427 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:48,427 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:48,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:48,432 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:48,432 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:48,432 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:48,432 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:48,432 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:48,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:48,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,435 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,436 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,440 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,440 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,440 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:48,440 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,440 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:48,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:48,440 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,440 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,440 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,440 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:48,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:48,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:48,441 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:48,441 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:48,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:48,541 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:48,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:48,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:48,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:48,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:48,556 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:48,556 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,556 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:48,556 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:48,557 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,557 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,559 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:48,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:48,560 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,560 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:48,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:48,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:48,564 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:48,564 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,564 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:48,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:48,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360948577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:48,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360948578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:48,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:48,580 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,580 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:48,580 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:48,580 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:48,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:48,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:48,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:48,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,582 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,583 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,583 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:48,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60240dd2[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:48,593 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:48,593 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:48,595 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,595 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:48,596 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:48,599 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:48,599 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:48,599 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:48,599 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:48,599 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:48,602 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,602 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:48,602 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:48,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:48,603 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:48,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:48,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:48,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:48,607 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:48,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:48,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:48,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:48,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:48,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,607 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:48,608 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:48,696 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/f20964d95b6a472fa600226c2bda83db 2023-04-01 14:54:48,700 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/e4edf08d52cf4286a8d380c797fbb989 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e4edf08d52cf4286a8d380c797fbb989 2023-04-01 14:54:48,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e4edf08d52cf4286a8d380c797fbb989, entries=150, sequenceid=302, filesize=11.9 K 2023-04-01 14:54:48,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/e0a13db1df324174a1e3c424dc24d498 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/e0a13db1df324174a1e3c424dc24d498 2023-04-01 14:54:48,708 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:48,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:48,712 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/e0a13db1df324174a1e3c424dc24d498, entries=150, sequenceid=302, filesize=11.9 K 2023-04-01 14:54:48,713 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/f20964d95b6a472fa600226c2bda83db as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f20964d95b6a472fa600226c2bda83db 2023-04-01 14:54:48,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:48,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:48,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:48,715 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:48,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:48,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:48,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,718 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f20964d95b6a472fa600226c2bda83db, entries=150, sequenceid=302, filesize=11.9 K 2023-04-01 14:54:48,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5326183795415453ea78c16f1f182c7d in 857ms, sequenceid=302, compaction requested=true 2023-04-01 14:54:48,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:48,719 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:48,719 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:48,719 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:54:48,719 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:54:48,719 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:48,720 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,720 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:48,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:48,721 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:48,721 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,721 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63548 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:54:48,721 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/A is initiating minor compaction (all files) 2023-04-01 14:54:48,721 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61108 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:54:48,721 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/A in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,721 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/B is initiating minor compaction (all files) 2023-04-01 14:54:48,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:48,721 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/B in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,721 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/1d5b2eef22034e96bd762f9104ec25bb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/79a5be2ce6734d67bda232a14038761b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/69c790d897f94ae9be33e86ab9c115d4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e6773508d0d44712af6555cb59476424, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e4edf08d52cf4286a8d380c797fbb989] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=62.1 K 2023-04-01 14:54:48,721 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/cc666f9ad04448beb331868bd6933dab, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/8de146800bb449b78dcae7824c2b9190, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1a16b22ed4ba4e9986d99404b39d9bac, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/063813620c0c463b8928ada40fe87294, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/e0a13db1df324174a1e3c424dc24d498] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=59.7 K 2023-04-01 14:54:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:48,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:48,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:48,722 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cc666f9ad04448beb331868bd6933dab, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680360882572 2023-04-01 14:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:48,722 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 1d5b2eef22034e96bd762f9104ec25bb, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680360882572 2023-04-01 14:54:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:48,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:48,723 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 8de146800bb449b78dcae7824c2b9190, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680360883876 2023-04-01 14:54:48,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:48,723 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 79a5be2ce6734d67bda232a14038761b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680360883876 2023-04-01 14:54:48,723 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-01 14:54:48,723 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1a16b22ed4ba4e9986d99404b39d9bac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680360885047 2023-04-01 14:54:48,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:48,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:48,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:48,723 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 69c790d897f94ae9be33e86ab9c115d4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680360885045 2023-04-01 14:54:48,723 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 063813620c0c463b8928ada40fe87294, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680360885707 2023-04-01 14:54:48,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:48,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:48,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:48,724 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting e6773508d0d44712af6555cb59476424, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680360885707 2023-04-01 14:54:48,724 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e0a13db1df324174a1e3c424dc24d498, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680360887855 2023-04-01 14:54:48,724 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting e4edf08d52cf4286a8d380c797fbb989, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680360887855 2023-04-01 14:54:48,743 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#B#compaction#256 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:48,744 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/543dc2ffbc8a49fb9dd0afe750fb3bbe 2023-04-01 14:54:48,745 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#A#compaction#257 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:48,768 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/51909c0a9c84442dad9d50875fc6f714 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/51909c0a9c84442dad9d50875fc6f714 2023-04-01 14:54:48,772 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5326183795415453ea78c16f1f182c7d/B of 5326183795415453ea78c16f1f182c7d into 51909c0a9c84442dad9d50875fc6f714(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:48,772 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:48,772 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/B, priority=11, startTime=1680360888719; duration=0sec 2023-04-01 14:54:48,772 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:48,772 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:54:48,775 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61108 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:54:48,775 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/C is initiating minor compaction (all files) 2023-04-01 14:54:48,775 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/C in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:48,775 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e99527f73a894d42a124a915a7a3a406, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a959a00641284f90b9024d10fbafd7dd, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f9b5982cae604683809b82fc4f34f0b5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4375f41e92c045cfbb273c9db766b1d9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f20964d95b6a472fa600226c2bda83db] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=59.7 K 2023-04-01 14:54:48,775 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e99527f73a894d42a124a915a7a3a406, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680360882572 2023-04-01 14:54:48,776 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a959a00641284f90b9024d10fbafd7dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680360883876 2023-04-01 14:54:48,776 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f9b5982cae604683809b82fc4f34f0b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680360885047 2023-04-01 14:54:48,780 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/51d5656887764e5fb4f78bb0c4bd9a04 2023-04-01 14:54:48,783 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 4375f41e92c045cfbb273c9db766b1d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680360885707 2023-04-01 14:54:48,784 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f20964d95b6a472fa600226c2bda83db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680360887855 2023-04-01 14:54:48,798 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#C#compaction#260 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:49,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:49,082 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:49,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:49,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360949092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:49,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360949093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,178 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/b9ab9045ccca499a9cb288e61cd301b6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/b9ab9045ccca499a9cb288e61cd301b6 2023-04-01 14:54:49,182 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5326183795415453ea78c16f1f182c7d/A of 5326183795415453ea78c16f1f182c7d into b9ab9045ccca499a9cb288e61cd301b6(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:49,182 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:49,182 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/A, priority=11, startTime=1680360888719; duration=0sec 2023-04-01 14:54:49,182 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:49,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:49,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360949197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:49,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360949197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,206 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/05deecb45af24fc296bbc4e08c01da02 2023-04-01 14:54:49,211 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/543dc2ffbc8a49fb9dd0afe750fb3bbe as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/543dc2ffbc8a49fb9dd0afe750fb3bbe 2023-04-01 14:54:49,214 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/543dc2ffbc8a49fb9dd0afe750fb3bbe, entries=150, sequenceid=355, filesize=11.9 K 2023-04-01 14:54:49,215 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/51d5656887764e5fb4f78bb0c4bd9a04 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/51d5656887764e5fb4f78bb0c4bd9a04 2023-04-01 14:54:49,219 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/f99c658e03b14f76afc4ee812dd7cb92 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f99c658e03b14f76afc4ee812dd7cb92 2023-04-01 14:54:49,220 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/51d5656887764e5fb4f78bb0c4bd9a04, entries=150, sequenceid=355, filesize=11.9 K 2023-04-01 14:54:49,221 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/05deecb45af24fc296bbc4e08c01da02 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/05deecb45af24fc296bbc4e08c01da02 2023-04-01 14:54:49,224 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5326183795415453ea78c16f1f182c7d/C of 5326183795415453ea78c16f1f182c7d into f99c658e03b14f76afc4ee812dd7cb92(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:49,224 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:49,224 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/C, priority=11, startTime=1680360888719; duration=0sec 2023-04-01 14:54:49,224 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:49,225 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/05deecb45af24fc296bbc4e08c01da02, entries=150, sequenceid=355, filesize=11.9 K 2023-04-01 14:54:49,227 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=46.96 KB/48090 for 5326183795415453ea78c16f1f182c7d in 504ms, sequenceid=355, compaction requested=false 2023-04-01 14:54:49,227 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:49,227 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:49,227 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:49,227 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:49,228 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,228 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:49,228 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:49,229 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,229 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:49,230 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:49,230 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:49,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:49,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@515ee700[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:49,231 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:49,231 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:49,233 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:49,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:49,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:49,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:49,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:49,233 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,233 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:49,235 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:49,235 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,235 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:49,235 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:49,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:49,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:49,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:49,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:49,237 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:49,237 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:49,237 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:49,237 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:49,237 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:49,239 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:49,239 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:49,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:49,239 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:49,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:49,240 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:49,241 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,243 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,243 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:49,243 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:49,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:49,243 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:49,243 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:49,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:49,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:49,243 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:49,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:49,243 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,243 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:49,243 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:49,244 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:49,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:49,243 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:49,244 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:49,344 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:49,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:49,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:49,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:49,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:49,347 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:49,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:49,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:49,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:49,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:49,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:49,350 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:49,350 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:49,350 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,350 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:49,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:49,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:49,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:49,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:49,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:49,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:49,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:49,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:49,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:49,352 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:49,352 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:49,352 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:54:49,352 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:49,352 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:49,352 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:49,352 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:49,352 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:49,352 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:49,367 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/31a572fd817147079384218dd79b41f8 2023-04-01 14:54:49,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:49,400 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:49,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360889417, max timestamp=1680360889422Num uniques 859; 2023-04-01 14:54:49,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:54:49,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360889417, max timestamp=1680360889422Num uniques 859; 2023-04-01 14:54:49,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360889417, max timestamp=1680360889422Num uniques 859; 2023-04-01 14:54:49,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:54:49,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:54:49,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:49,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360949446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:49,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360949446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:49,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:49,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360949550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360949550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:49,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:49,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360949754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360949754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:49,779 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/4531393e38e84cdbbe834c88452c01bc 2023-04-01 14:54:50,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:50,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:50,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360950057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360950058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,189 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/63d7992cfedb4b01873ab44d8f445dc0 2023-04-01 14:54:50,193 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/31a572fd817147079384218dd79b41f8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/31a572fd817147079384218dd79b41f8 2023-04-01 14:54:50,197 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/31a572fd817147079384218dd79b41f8, entries=100, sequenceid=368, filesize=9.5 K 2023-04-01 14:54:50,197 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/4531393e38e84cdbbe834c88452c01bc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/4531393e38e84cdbbe834c88452c01bc 2023-04-01 14:54:50,201 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/4531393e38e84cdbbe834c88452c01bc, entries=100, sequenceid=368, filesize=9.5 K 2023-04-01 14:54:50,201 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/63d7992cfedb4b01873ab44d8f445dc0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/63d7992cfedb4b01873ab44d8f445dc0 2023-04-01 14:54:50,205 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/63d7992cfedb4b01873ab44d8f445dc0, entries=100, sequenceid=368, filesize=9.5 K 2023-04-01 14:54:50,205 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 5326183795415453ea78c16f1f182c7d in 853ms, sequenceid=368, compaction requested=true 2023-04-01 14:54:50,205 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:50,205 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:50,206 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:50,206 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:50,206 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,206 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:50,206 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:50,207 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,207 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:50,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:50,208 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,208 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:50,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:50,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4498e3[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:50,209 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:50,209 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,210 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:50,211 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:50,211 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,211 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:50,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:50,216 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:50,217 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:50,217 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:50,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:50,219 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:50,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:50,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:50,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:50,219 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:50,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:50,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:50,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:50,221 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,222 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,225 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,225 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:50,225 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,225 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:50,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:50,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:50,225 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:50,225 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:50,225 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,225 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,225 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:50,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:50,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:50,225 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:50,225 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,225 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,325 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:50,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:50,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:50,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:50,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:50,328 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:50,328 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:50,328 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:50,329 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:50,329 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,329 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,331 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:50,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:50,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:50,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:50,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:50,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:50,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:50,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:50,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:50,333 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-01 14:54:50,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:50,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:50,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:50,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:50,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:50,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:50,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:50,564 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:50,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:50,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360950571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:50,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360950572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:50,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360950675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:50,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360950675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,740 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/21e03570caca4506892da30d4a02795a 2023-04-01 14:54:50,751 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/da20177158954b56884abc0bd8b1286e 2023-04-01 14:54:50,761 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/4ebc1f3630774cedb14aad6ad7e938bc 2023-04-01 14:54:50,773 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/21e03570caca4506892da30d4a02795a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/21e03570caca4506892da30d4a02795a 2023-04-01 14:54:50,778 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/21e03570caca4506892da30d4a02795a, entries=150, sequenceid=405, filesize=11.9 K 2023-04-01 14:54:50,779 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/da20177158954b56884abc0bd8b1286e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/da20177158954b56884abc0bd8b1286e 2023-04-01 14:54:50,782 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/da20177158954b56884abc0bd8b1286e, entries=150, sequenceid=405, filesize=11.9 K 2023-04-01 14:54:50,784 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/4ebc1f3630774cedb14aad6ad7e938bc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4ebc1f3630774cedb14aad6ad7e938bc 2023-04-01 14:54:50,787 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4ebc1f3630774cedb14aad6ad7e938bc, entries=150, sequenceid=405, filesize=11.9 K 2023-04-01 14:54:50,788 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 5326183795415453ea78c16f1f182c7d in 455ms, sequenceid=405, compaction requested=true 2023-04-01 14:54:50,788 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:50,788 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:50,790 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,791 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,791 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:50,795 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:50,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:50,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@329a0934[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:50,797 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:50,797 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,799 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:50,799 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:50,799 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:50,799 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,799 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:50,800 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,800 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:50,800 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:50,800 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:50,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:50,803 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:50,803 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:50,803 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:50,803 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:50,803 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:50,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:50,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:50,807 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:50,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:50,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:50,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:50,811 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:50,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:50,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:50,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:50,811 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:50,812 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:50,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:50,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:54:50,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:50,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:50,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:50,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:50,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:50,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:50,912 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:50,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:50,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:50,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360890905, max timestamp=1680360890914Num uniques 859; 2023-04-01 14:54:50,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:50,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360890905, max timestamp=1680360890914Num uniques 859; 2023-04-01 14:54:50,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360890905, max timestamp=1680360890914Num uniques 859; 2023-04-01 14:54:50,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:50,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:50,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:54:50,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:50,916 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:50,916 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:50,916 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:50,916 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:50,917 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,917 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,918 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,918 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:50,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:50,918 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,918 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:50,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:50,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:50,919 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:50,920 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:50,920 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:50,920 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:50,920 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:50,922 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:50,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:50,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7210c828[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:50,924 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:50,924 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:50,927 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:50,928 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:50,930 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:50,930 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:50,930 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:50,930 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:50,930 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:50,931 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,931 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:50,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,931 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:50,932 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,933 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:50,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:50,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:50,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:50,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:50,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:50,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:50,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:50,935 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:50,936 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:50,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,936 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:50,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:50,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:50,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:50,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360950943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:50,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:50,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360950947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,036 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:51,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:51,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:51,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:51,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:51,039 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:51,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:51,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:51,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,042 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,042 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,042 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:51,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:51,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:51,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:51,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:51,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:51,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:51,043 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,043 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:51,044 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:51,044 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:51,045 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5450478d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:51,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:51,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,049 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:51,049 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,049 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:51,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:51,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360951048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:51,050 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360951050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:51,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:51,053 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:51,053 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:51,053 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:51,053 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:51,053 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:51,055 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,055 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,055 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,057 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,062 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:51,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:51,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,062 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,063 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:51,063 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:51,163 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:51,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:51,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:51,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:51,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:51,166 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:51,166 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,166 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:51,166 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:51,167 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,167 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,168 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,168 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,168 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,168 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:51,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:51,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:51,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:51,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:51,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:51,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:51,170 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,170 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:51,170 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:51,170 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:51,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:51,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:51,170 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:51,172 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,172 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,172 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:51,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fb84e64[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:51,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:51,175 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,178 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:51,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:51,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:51,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:51,179 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:51,183 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:51,183 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:51,183 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:51,183 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:51,183 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:51,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,189 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:51,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:51,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:51,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,193 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:51,194 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:51,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:51,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360951250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:51,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360951253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,294 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:51,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:51,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:51,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:51,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:51,297 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:51,297 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,297 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:51,297 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:51,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/6eb09658f0bc46fca68788c1fe5c9ef4 2023-04-01 14:54:51,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,301 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,301 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:51,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:51,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:51,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:51,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:51,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:51,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:51,304 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,304 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:51,304 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:51,305 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,305 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:51,305 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:51,305 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,305 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:51,305 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:51,307 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,307 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,307 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:51,308 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:51,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d09330e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:51,312 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:51,312 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,313 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:51,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:51,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:51,319 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:51,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:51,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:51,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:51,319 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:51,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,323 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,324 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,328 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,328 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:51,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:51,328 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,328 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:51,328 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:51,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,428 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:51,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:51,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:51,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:51,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:51,432 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:51,432 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,432 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:51,432 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:51,432 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,433 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,434 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,434 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,434 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,434 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:51,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:51,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:51,436 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,436 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:51,436 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:51,436 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,437 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,438 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,438 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:51,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29ace10f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:51,439 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:51,439 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,441 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:51,441 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,441 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:51,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:51,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,443 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:51,445 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:51,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:51,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:51,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:51,446 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:51,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,450 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:51,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:51,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:51,455 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,455 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:51,455 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:51,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,555 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:51,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:51,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:51,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:51,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360951555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:51,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360951556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:51,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:51,559 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:51,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:51,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:51,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,560 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,560 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,560 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,560 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:51,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:51,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:51,562 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,562 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:51,562 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:51,562 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:51,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:51,563 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,564 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,564 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:51,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f16b798[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:51,565 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:51,565 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,567 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,567 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,567 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:51,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,567 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:51,567 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:51,567 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:51,568 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,568 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:51,568 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,569 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:51,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:51,570 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:51,570 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:51,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:51,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:51,571 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:51,572 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,572 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,572 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,573 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,575 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,575 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,576 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,576 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:51,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:51,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,576 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,576 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,576 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:51,576 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,576 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,576 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:51,576 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:51,576 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,676 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:51,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:51,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:51,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:51,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:51,679 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:51,680 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,681 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:51,681 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:51,681 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,681 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,683 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:51,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:51,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:51,685 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,685 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:51,685 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:51,685 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:51,687 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:51,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2201d70[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:51,689 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:51,689 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,690 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:51,690 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,690 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:51,692 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:51,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:51,695 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:51,695 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:51,695 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:51,695 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:51,695 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:51,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,699 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,700 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,703 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:51,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:51,704 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:51,704 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,723 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/7f94dac7e5634375acf582ee819f1e86 2023-04-01 14:54:51,804 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:51,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:51,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:51,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:51,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:51,808 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:51,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:51,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:51,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,809 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,809 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,810 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,810 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:51,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:51,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:51,811 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,811 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:51,811 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:51,811 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:51,811 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:51,813 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,813 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,813 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,813 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:51,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a5df79[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:51,815 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:51,815 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,817 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:51,817 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,817 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:51,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:51,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:51,821 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:51,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:51,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:51,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:51,821 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:51,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,823 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,824 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,826 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,826 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,826 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,826 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,826 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:51,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:51,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:51,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,827 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:51,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,827 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:51,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,927 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:51,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:51,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:51,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:51,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:51,930 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:51,930 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,930 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:51,930 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:51,931 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,931 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,936 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,936 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,936 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:51,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:51,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:51,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:51,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:51,938 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,938 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:51,938 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:51,938 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:51,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:51,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:51,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:51,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:51,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,939 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,940 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,940 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:51,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58cff03b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:51,941 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:51,941 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,942 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:51,942 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,942 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:51,945 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:51,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:51,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:51,947 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:51,947 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:51,947 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:51,947 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:51,947 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:51,949 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,949 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,949 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:51,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:51,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:51,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:51,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:51,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:51,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:51,953 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:51,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:51,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:51,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:51,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:51,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:51,954 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:51,954 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:52,054 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:52,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:52,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:52,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:52,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:52,057 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:52,057 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:52,057 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:52,057 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:52,057 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:52,057 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:52,059 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:52,059 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:52,059 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,059 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:52,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:52,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:52,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:52,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:52,060 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:52,060 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:52,060 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:52,060 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:52,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:52,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360952059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:52,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360952061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,063 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:52,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:52,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12af26cd[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:52,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:52,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:52,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:52,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:52,066 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:52,066 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:52,066 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:52,066 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,066 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:52,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:52,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:52,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:52,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:52,070 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:52,070 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:52,070 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:52,070 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:52,070 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:52,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:52,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:52,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:52,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:52,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:52,072 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,073 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,076 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,076 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:52,076 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:52,076 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:52,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:52,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:52,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:52,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:52,076 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:52,077 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:52,076 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,077 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:52,077 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:52,077 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:52,077 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:52,077 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:52,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:52,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/fee7b90339904eb08790c9051cc51e22 2023-04-01 14:54:52,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/6eb09658f0bc46fca68788c1fe5c9ef4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/6eb09658f0bc46fca68788c1fe5c9ef4 2023-04-01 14:54:52,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/6eb09658f0bc46fca68788c1fe5c9ef4, entries=150, sequenceid=416, filesize=11.9 K 2023-04-01 14:54:52,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/7f94dac7e5634375acf582ee819f1e86 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/7f94dac7e5634375acf582ee819f1e86 2023-04-01 14:54:52,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/7f94dac7e5634375acf582ee819f1e86, entries=150, sequenceid=416, filesize=11.9 K 2023-04-01 14:54:52,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/fee7b90339904eb08790c9051cc51e22 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fee7b90339904eb08790c9051cc51e22 2023-04-01 14:54:52,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fee7b90339904eb08790c9051cc51e22, entries=150, sequenceid=416, filesize=11.9 K 2023-04-01 14:54:52,173 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5326183795415453ea78c16f1f182c7d in 1292ms, sequenceid=416, compaction requested=true 2023-04-01 14:54:52,173 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:52,173 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:52,173 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:52,173 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:52,173 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:54:52,173 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:54:52,175 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59134 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:54:52,175 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/B is initiating minor compaction (all files) 2023-04-01 14:54:52,175 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59134 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:54:52,175 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/B in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:52,175 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/A is initiating minor compaction (all files) 2023-04-01 14:54:52,175 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/51909c0a9c84442dad9d50875fc6f714, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/51d5656887764e5fb4f78bb0c4bd9a04, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/4531393e38e84cdbbe834c88452c01bc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/da20177158954b56884abc0bd8b1286e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/7f94dac7e5634375acf582ee819f1e86] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=57.7 K 2023-04-01 14:54:52,175 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/A in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:52,175 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/b9ab9045ccca499a9cb288e61cd301b6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/543dc2ffbc8a49fb9dd0afe750fb3bbe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/31a572fd817147079384218dd79b41f8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/21e03570caca4506892da30d4a02795a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/6eb09658f0bc46fca68788c1fe5c9ef4] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=57.7 K 2023-04-01 14:54:52,176 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 51909c0a9c84442dad9d50875fc6f714, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680360887855 2023-04-01 14:54:52,176 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 51d5656887764e5fb4f78bb0c4bd9a04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680360887962 2023-04-01 14:54:52,176 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting b9ab9045ccca499a9cb288e61cd301b6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680360887855 2023-04-01 14:54:52,177 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 4531393e38e84cdbbe834c88452c01bc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1680360889091 2023-04-01 14:54:52,177 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 543dc2ffbc8a49fb9dd0afe750fb3bbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680360887962 2023-04-01 14:54:52,177 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting da20177158954b56884abc0bd8b1286e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680360889441 2023-04-01 14:54:52,177 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 31a572fd817147079384218dd79b41f8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1680360889091 2023-04-01 14:54:52,177 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:52,177 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 7f94dac7e5634375acf582ee819f1e86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680360890567 2023-04-01 14:54:52,177 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 21e03570caca4506892da30d4a02795a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680360889441 2023-04-01 14:54:52,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:52,178 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 6eb09658f0bc46fca68788c1fe5c9ef4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680360890567 2023-04-01 14:54:52,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:52,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:52,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:52,183 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:52,183 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:52,183 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:52,183 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:52,184 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:52,184 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:52,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:52,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:52,185 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:52,186 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:52,186 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:52,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:52,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:52,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:52,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:52,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:52,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:52,196 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-01 14:54:52,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:52,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:52,197 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:52,197 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:52,197 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:52,197 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:54:52,202 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#A#compaction#270 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:52,203 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#B#compaction#271 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:52,251 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/25b2dbb373ce4e54809494bdcb4d7fe1 2023-04-01 14:54:52,633 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/19ce98178aa040e2a11745c80fb44522 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/19ce98178aa040e2a11745c80fb44522 2023-04-01 14:54:52,637 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5326183795415453ea78c16f1f182c7d/A of 5326183795415453ea78c16f1f182c7d into 19ce98178aa040e2a11745c80fb44522(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:52,637 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:52,638 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/A, priority=11, startTime=1680360892173; duration=0sec 2023-04-01 14:54:52,638 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:52,638 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:54:52,640 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59134 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:54:52,640 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/C is initiating minor compaction (all files) 2023-04-01 14:54:52,640 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/C in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:52,640 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f99c658e03b14f76afc4ee812dd7cb92, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/05deecb45af24fc296bbc4e08c01da02, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/63d7992cfedb4b01873ab44d8f445dc0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4ebc1f3630774cedb14aad6ad7e938bc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fee7b90339904eb08790c9051cc51e22] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=57.7 K 2023-04-01 14:54:52,640 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting f99c658e03b14f76afc4ee812dd7cb92, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680360887855 2023-04-01 14:54:52,641 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 05deecb45af24fc296bbc4e08c01da02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1680360887962 2023-04-01 14:54:52,641 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 63d7992cfedb4b01873ab44d8f445dc0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1680360889091 2023-04-01 14:54:52,642 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 4ebc1f3630774cedb14aad6ad7e938bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680360889441 2023-04-01 14:54:52,643 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting fee7b90339904eb08790c9051cc51e22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680360890567 2023-04-01 14:54:52,643 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/97605c4ca67a4ef486c32789d0e03013 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/97605c4ca67a4ef486c32789d0e03013 2023-04-01 14:54:52,650 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5326183795415453ea78c16f1f182c7d/B of 5326183795415453ea78c16f1f182c7d into 97605c4ca67a4ef486c32789d0e03013(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:52,650 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:52,650 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/B, priority=11, startTime=1680360892173; duration=0sec 2023-04-01 14:54:52,650 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:52,660 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#C#compaction#274 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:52,664 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/fcf2fefeaf694f2abd5224ecc124fa07 2023-04-01 14:54:53,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:53,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:53,076 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/6b45d751cda2470687ac7a5ae0c36f9e 2023-04-01 14:54:53,080 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/25b2dbb373ce4e54809494bdcb4d7fe1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/25b2dbb373ce4e54809494bdcb4d7fe1 2023-04-01 14:54:53,082 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/29f6d375ebd34270802351cbda637a37 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/29f6d375ebd34270802351cbda637a37 2023-04-01 14:54:53,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360953081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360953081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,085 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/25b2dbb373ce4e54809494bdcb4d7fe1, entries=150, sequenceid=452, filesize=11.9 K 2023-04-01 14:54:53,085 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/fcf2fefeaf694f2abd5224ecc124fa07 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/fcf2fefeaf694f2abd5224ecc124fa07 2023-04-01 14:54:53,089 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/fcf2fefeaf694f2abd5224ecc124fa07, entries=150, sequenceid=452, filesize=11.9 K 2023-04-01 14:54:53,090 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/6b45d751cda2470687ac7a5ae0c36f9e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/6b45d751cda2470687ac7a5ae0c36f9e 2023-04-01 14:54:53,091 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5326183795415453ea78c16f1f182c7d/C of 5326183795415453ea78c16f1f182c7d into 29f6d375ebd34270802351cbda637a37(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:53,091 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:53,091 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/C, priority=11, startTime=1680360892173; duration=0sec 2023-04-01 14:54:53,091 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:53,097 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/6b45d751cda2470687ac7a5ae0c36f9e, entries=150, sequenceid=452, filesize=11.9 K 2023-04-01 14:54:53,099 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 5326183795415453ea78c16f1f182c7d in 903ms, sequenceid=452, compaction requested=false 2023-04-01 14:54:53,100 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:53,100 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:53,100 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:53,102 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,102 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,102 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,102 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:53,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10e66275[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:53,104 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:53,104 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,105 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:53,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:53,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:53,106 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,106 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:53,108 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:53,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:53,110 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:53,110 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:53,111 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:53,111 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:53,111 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:53,112 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,112 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,112 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:53,113 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,114 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,116 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,117 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,117 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,117 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:53,117 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,117 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,117 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,117 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:53,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:53,117 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:53,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:53,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:53,117 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:53,117 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:53,117 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:53,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:53,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:53,187 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 14:54:53,187 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:53,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:53,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:53,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:53,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:53,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:53,217 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:53,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:53,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:53,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:53,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:53,223 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:53,224 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:53,224 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:53,224 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:53,224 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,224 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,225 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:53,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:53,225 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,226 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,226 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:53,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:53,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/bbd8d2d27731469cbe944f452a75aacf 2023-04-01 14:54:53,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360953225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360953227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:53,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:53,236 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:53,236 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:53,236 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:53,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:53,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:53,244 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,244 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:53,244 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:53,244 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:53,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:53,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:53,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:53,249 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,249 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,249 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,249 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:53,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28002907[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:53,250 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:53,251 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:53,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:53,253 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:53,253 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:53,253 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:53,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,253 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/71c3067cae584b4aaaa0713b07f9935f 2023-04-01 14:54:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:53,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:53,257 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:53,257 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:53,257 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:53,257 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:53,257 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:53,259 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,259 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,259 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:53,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,260 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,262 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:53,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:53,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:53,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,265 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:53,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:53,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:53,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:53,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:53,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,266 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:53,266 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:53,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360953328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360953332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,366 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:53,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:53,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:53,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:53,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:53,369 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:53,369 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:53,369 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:53,369 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:53,370 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,370 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,371 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:53,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:53,371 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,372 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,372 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:53,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:53,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:53,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:53,373 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:53,373 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:53,373 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:53,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:53,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:53,375 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,375 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:53,375 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:53,375 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:53,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:53,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:53,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:53,376 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,376 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,377 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,377 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:53,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f695a14[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:53,379 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:53,379 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,380 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:53,381 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,381 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:53,383 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:53,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,383 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,384 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:53,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:53,398 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:53,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:53,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:53,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:53,398 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:53,401 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,401 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,401 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:53,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,402 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,404 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:53,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:53,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:53,407 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:53,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:53,408 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,408 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,408 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,408 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:53,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:53,408 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:53,408 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:53,508 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:53,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:53,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:53,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:53,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:53,512 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:53,512 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:53,512 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:53,512 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:53,512 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,512 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,514 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:53,514 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:53,514 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,514 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:53,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:53,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:53,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:53,516 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:53,516 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:53,516 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,516 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:53,516 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:53,516 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,516 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:53,516 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:53,516 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,516 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:53,516 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:53,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:53,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@641b9c7a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:53,525 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:53,525 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,526 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:53,526 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,526 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:53,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:53,530 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:53,530 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:53,530 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:53,530 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:53,530 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:53,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360953530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:53,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:53,533 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,534 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360953534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:53,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:53,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:53,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:53,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:53,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:53,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:53,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:53,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:53,539 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:53,539 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:53,639 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:53,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:53,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:53,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:53,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:53,642 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:53,642 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:53,642 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:53,642 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:53,643 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,643 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,644 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,644 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:53,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:53,644 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,644 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:53,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:53,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:53,646 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,646 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:53,646 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:53,646 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:53,646 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:53,648 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47cdaf46[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:53,649 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:53,649 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,650 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:53,651 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:53,651 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,651 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:53,656 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:53,659 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:53,659 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:53,659 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:53,659 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:53,659 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:53,660 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,660 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:53,661 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:53,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:53,662 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,663 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:53,665 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:53,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:53,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:53,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:53,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:53,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,666 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,666 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,666 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:53,666 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,666 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:53,666 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:53,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:53,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/4e1ee337dd5c4d108769ed97be658192 2023-04-01 14:54:53,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:53,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/bbd8d2d27731469cbe944f452a75aacf as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/bbd8d2d27731469cbe944f452a75aacf 2023-04-01 14:54:53,677 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/bbd8d2d27731469cbe944f452a75aacf, entries=150, sequenceid=467, filesize=11.9 K 2023-04-01 14:54:53,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/71c3067cae584b4aaaa0713b07f9935f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/71c3067cae584b4aaaa0713b07f9935f 2023-04-01 14:54:53,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/71c3067cae584b4aaaa0713b07f9935f, entries=150, sequenceid=467, filesize=11.9 K 2023-04-01 14:54:53,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/4e1ee337dd5c4d108769ed97be658192 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4e1ee337dd5c4d108769ed97be658192 2023-04-01 14:54:53,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4e1ee337dd5c4d108769ed97be658192, entries=150, sequenceid=467, filesize=11.9 K 2023-04-01 14:54:53,691 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5326183795415453ea78c16f1f182c7d in 504ms, sequenceid=467, compaction requested=true 2023-04-01 14:54:53,691 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:53,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:53,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 14:54:53,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-01 14:54:53,692 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:53,692 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:53,693 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:53,693 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/A is initiating minor compaction (all files) 2023-04-01 14:54:53,693 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/A in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,693 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/19ce98178aa040e2a11745c80fb44522, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/25b2dbb373ce4e54809494bdcb4d7fe1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/bbd8d2d27731469cbe944f452a75aacf] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=36.5 K 2023-04-01 14:54:53,693 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:53,693 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/C is initiating minor compaction (all files) 2023-04-01 14:54:53,693 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/C in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,694 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/29f6d375ebd34270802351cbda637a37, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/6b45d751cda2470687ac7a5ae0c36f9e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4e1ee337dd5c4d108769ed97be658192] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=36.5 K 2023-04-01 14:54:53,694 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 29f6d375ebd34270802351cbda637a37, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680360890567 2023-04-01 14:54:53,694 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 19ce98178aa040e2a11745c80fb44522, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680360890567 2023-04-01 14:54:53,694 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6b45d751cda2470687ac7a5ae0c36f9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680360890937 2023-04-01 14:54:53,695 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 25b2dbb373ce4e54809494bdcb4d7fe1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680360890937 2023-04-01 14:54:53,695 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 4e1ee337dd5c4d108769ed97be658192, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680360893079 2023-04-01 14:54:53,695 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting bbd8d2d27731469cbe944f452a75aacf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680360893079 2023-04-01 14:54:53,705 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#C#compaction#279 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:53,707 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#A#compaction#280 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:53,738 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/c87f5bb30299474a8dfb0e8b670a70a0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/c87f5bb30299474a8dfb0e8b670a70a0 2023-04-01 14:54:53,741 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5326183795415453ea78c16f1f182c7d/C of 5326183795415453ea78c16f1f182c7d into c87f5bb30299474a8dfb0e8b670a70a0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:53,741 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:53,741 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/C, priority=13, startTime=1680360893691; duration=0sec 2023-04-01 14:54:53,742 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:53,742 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:53,743 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:53,743 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/B is initiating minor compaction (all files) 2023-04-01 14:54:53,743 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/B in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,743 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/97605c4ca67a4ef486c32789d0e03013, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/fcf2fefeaf694f2abd5224ecc124fa07, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/71c3067cae584b4aaaa0713b07f9935f] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=36.5 K 2023-04-01 14:54:53,743 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 97605c4ca67a4ef486c32789d0e03013, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680360890567 2023-04-01 14:54:53,743 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting fcf2fefeaf694f2abd5224ecc124fa07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680360890937 2023-04-01 14:54:53,744 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 71c3067cae584b4aaaa0713b07f9935f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680360893079 2023-04-01 14:54:53,749 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#B#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:53,766 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:53,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:53,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:54:53,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:54:53,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:54:53,769 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:54:53,769 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:53,769 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:54:53,769 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:54:53,769 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,769 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,770 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:53,770 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:53,771 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,771 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:54:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:54:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:53,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:54:53,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:54:53,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:54:53,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:54:53,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:54:53,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:54:53,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:54:53,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:54:53,772 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:53,773 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. started... 2023-04-01 14:54:53,773 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 14:54:53,773 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:53,773 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:53,773 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:53,773 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:53,773 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:53,773 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:53,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:53,835 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. as already flushing 2023-04-01 14:54:53,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360953850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360953851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:53,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360953954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:53,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360953954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:54,135 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/cc74b1a47f6f4435b41d958aa34b578b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/cc74b1a47f6f4435b41d958aa34b578b 2023-04-01 14:54:54,139 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5326183795415453ea78c16f1f182c7d/A of 5326183795415453ea78c16f1f182c7d into cc74b1a47f6f4435b41d958aa34b578b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:54,139 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:54,139 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/A, priority=13, startTime=1680360893691; duration=0sec 2023-04-01 14:54:54,139 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:54,158 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/a414cd37d5ff4257b8b4c4c2a0b63827 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/a414cd37d5ff4257b8b4c4c2a0b63827 2023-04-01 14:54:54,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:54,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360954158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:54,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:54,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360954158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:54,162 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5326183795415453ea78c16f1f182c7d/B of 5326183795415453ea78c16f1f182c7d into a414cd37d5ff4257b8b4c4c2a0b63827(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:54,162 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:54,162 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/B, priority=13, startTime=1680360893691; duration=0sec 2023-04-01 14:54:54,163 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:54,186 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/95c307e89e014c87b719e6cd1a157167 2023-04-01 14:54:54,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:54,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:54,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360954462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:54,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360954462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:54,595 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/1b5ec99d283d492580dd6b48a29366c4 2023-04-01 14:54:54,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:54,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60112 deadline: 1680360954623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:54,625 DEBUG [Thread-1536] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18229 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:54,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:54,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60152 deadline: 1680360954641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:54,643 DEBUG [Thread-1538] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18248 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:54,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:54,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60102 deadline: 1680360954648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:54,650 DEBUG [Thread-1542] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18259 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:54:54,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:54,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60138 deadline: 1680360954966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:54,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:54:54,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60128 deadline: 1680360954966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,007 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/0b94ed70ce254228a26cb989ada08014 2023-04-01 14:54:55,011 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/95c307e89e014c87b719e6cd1a157167 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95c307e89e014c87b719e6cd1a157167 2023-04-01 14:54:55,015 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95c307e89e014c87b719e6cd1a157167, entries=150, sequenceid=488, filesize=11.9 K 2023-04-01 14:54:55,017 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/1b5ec99d283d492580dd6b48a29366c4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1b5ec99d283d492580dd6b48a29366c4 2023-04-01 14:54:55,021 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1b5ec99d283d492580dd6b48a29366c4, entries=150, sequenceid=488, filesize=11.9 K 2023-04-01 14:54:55,022 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/0b94ed70ce254228a26cb989ada08014 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/0b94ed70ce254228a26cb989ada08014 2023-04-01 14:54:55,025 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/0b94ed70ce254228a26cb989ada08014, entries=150, sequenceid=488, filesize=11.9 K 2023-04-01 14:54:55,026 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 5326183795415453ea78c16f1f182c7d in 1253ms, sequenceid=488, compaction requested=false 2023-04-01 14:54:55,026 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:55,026 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:55,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:54:55,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:54:55,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:54:55,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:54:55,033 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,033 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:55,033 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:55,033 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:54:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:55,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:55,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:54:55,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e2cfea9[Count = 0] remaining members to acquire global barrier 2023-04-01 14:54:55,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:54:55,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:55,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:55,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:55,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:54:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:54:55,036 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:54:55,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:54:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:54:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:55,037 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:54:55,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:54:55,040 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:54:55,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:54:55,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:54:55,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:54:55,040 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:54:55,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:55,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:55,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:54:55,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:54:55,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:55,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:55,043 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:54:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:54:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:55,044 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:54:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:54:55,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,047 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:54:55,047 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,047 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:54:55,047 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:55,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:54:55,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:54:55,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:54:55,047 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:54:55,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:54:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:54:55,048 DEBUG [Thread-1546] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:54:55,048 DEBUG [Thread-1546] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:54:55,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:54:55,048 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:54:55,052 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:55,052 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:54:55,052 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:54:55,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:54:55,145 DEBUG [Thread-1553] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de064cb to 127.0.0.1:60465 2023-04-01 14:54:55,145 DEBUG [Thread-1553] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:55,145 DEBUG [Thread-1551] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:60465 2023-04-01 14:54:55,145 DEBUG [Thread-1551] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:55,145 DEBUG [Thread-1549] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:60465 2023-04-01 14:54:55,145 DEBUG [Thread-1549] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:55,146 DEBUG [Thread-1555] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54a39be8 to 127.0.0.1:60465 2023-04-01 14:54:55,146 DEBUG [Thread-1555] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:55,148 DEBUG [Thread-1546] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:54:55,148 DEBUG [Thread-1547] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:60465 2023-04-01 14:54:55,148 DEBUG [Thread-1547] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:55,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:54:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 5326183795415453ea78c16f1f182c7d 2023-04-01 14:54:55,973 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:54:55,974 DEBUG [Thread-1544] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08cd7cb0 to 127.0.0.1:60465 2023-04-01 14:54:55,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:54:55,974 DEBUG [Thread-1544] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:55,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:55,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:54:55,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:55,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:54:55,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:54:55,978 DEBUG [Thread-1540] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:60465 2023-04-01 14:54:55,978 DEBUG [Thread-1540] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:54:56,380 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/28983e6b5fd24518b5e8987b7246c82a 2023-04-01 14:54:56,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/3cb7a15f79e6444a8d8097aff7698346 2023-04-01 14:54:57,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/790cb8c58acb4a5face0eb1f0866da34 2023-04-01 14:54:57,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/28983e6b5fd24518b5e8987b7246c82a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/28983e6b5fd24518b5e8987b7246c82a 2023-04-01 14:54:57,205 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/28983e6b5fd24518b5e8987b7246c82a, entries=100, sequenceid=504, filesize=9.5 K 2023-04-01 14:54:57,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/3cb7a15f79e6444a8d8097aff7698346 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cb7a15f79e6444a8d8097aff7698346 2023-04-01 14:54:57,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cb7a15f79e6444a8d8097aff7698346, entries=100, sequenceid=504, filesize=9.5 K 2023-04-01 14:54:57,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/790cb8c58acb4a5face0eb1f0866da34 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/790cb8c58acb4a5face0eb1f0866da34 2023-04-01 14:54:57,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/790cb8c58acb4a5face0eb1f0866da34, entries=100, sequenceid=504, filesize=9.5 K 2023-04-01 14:54:57,213 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=6.71 KB/6870 for 5326183795415453ea78c16f1f182c7d in 1240ms, sequenceid=504, compaction requested=true 2023-04-01 14:54:57,214 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:57,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:57,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:57,214 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:57,214 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:57,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:57,215 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:57,215 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/A is initiating minor compaction (all files) 2023-04-01 14:54:57,215 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:57,215 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/A in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:57,215 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/B is initiating minor compaction (all files) 2023-04-01 14:54:57,215 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/cc74b1a47f6f4435b41d958aa34b578b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95c307e89e014c87b719e6cd1a157167, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/28983e6b5fd24518b5e8987b7246c82a] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=34.2 K 2023-04-01 14:54:57,215 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/B in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:57,215 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/a414cd37d5ff4257b8b4c4c2a0b63827, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1b5ec99d283d492580dd6b48a29366c4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cb7a15f79e6444a8d8097aff7698346] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=34.2 K 2023-04-01 14:54:57,215 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting cc74b1a47f6f4435b41d958aa34b578b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680360893079 2023-04-01 14:54:57,215 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a414cd37d5ff4257b8b4c4c2a0b63827, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680360893079 2023-04-01 14:54:57,216 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 95c307e89e014c87b719e6cd1a157167, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1680360893203 2023-04-01 14:54:57,216 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1b5ec99d283d492580dd6b48a29366c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1680360893203 2023-04-01 14:54:57,216 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 28983e6b5fd24518b5e8987b7246c82a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680360893846 2023-04-01 14:54:57,216 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3cb7a15f79e6444a8d8097aff7698346, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680360893846 2023-04-01 14:54:57,222 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#B#compaction#288 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:57,222 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#A#compaction#289 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:57,630 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/f19b97ce339547b59641aa9a6d8b4b8e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/f19b97ce339547b59641aa9a6d8b4b8e 2023-04-01 14:54:57,630 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/156efed779b64b72bcc62d768eb371ff as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/156efed779b64b72bcc62d768eb371ff 2023-04-01 14:54:57,634 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5326183795415453ea78c16f1f182c7d/A of 5326183795415453ea78c16f1f182c7d into 156efed779b64b72bcc62d768eb371ff(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:57,634 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5326183795415453ea78c16f1f182c7d/B of 5326183795415453ea78c16f1f182c7d into f19b97ce339547b59641aa9a6d8b4b8e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:57,634 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:57,634 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:57,634 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/A, priority=13, startTime=1680360897214; duration=0sec 2023-04-01 14:54:57,634 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/B, priority=13, startTime=1680360897214; duration=0sec 2023-04-01 14:54:57,634 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:54:57,634 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:54:57,634 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:54:57,635 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:54:57,635 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 5326183795415453ea78c16f1f182c7d/C is initiating minor compaction (all files) 2023-04-01 14:54:57,635 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5326183795415453ea78c16f1f182c7d/C in TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:54:57,635 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/c87f5bb30299474a8dfb0e8b670a70a0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/0b94ed70ce254228a26cb989ada08014, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/790cb8c58acb4a5face0eb1f0866da34] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp, totalSize=34.2 K 2023-04-01 14:54:57,635 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting c87f5bb30299474a8dfb0e8b670a70a0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1680360893079 2023-04-01 14:54:57,636 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 0b94ed70ce254228a26cb989ada08014, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1680360893203 2023-04-01 14:54:57,636 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 790cb8c58acb4a5face0eb1f0866da34, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680360893846 2023-04-01 14:54:57,640 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5326183795415453ea78c16f1f182c7d#C#compaction#290 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:54:58,048 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/780665ef3c6b427b966fb99f64f913a1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/780665ef3c6b427b966fb99f64f913a1 2023-04-01 14:54:58,052 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5326183795415453ea78c16f1f182c7d/C of 5326183795415453ea78c16f1f182c7d into 780665ef3c6b427b966fb99f64f913a1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:54:58,052 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:54:58,052 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d., storeName=5326183795415453ea78c16f1f182c7d/C, priority=13, startTime=1680360897214; duration=0sec 2023-04-01 14:54:58,052 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:04,664 DEBUG [Thread-1536] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:60465 2023-04-01 14:55:04,664 DEBUG [Thread-1536] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:04,690 DEBUG [Thread-1542] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7810a231 to 127.0.0.1:60465 2023-04-01 14:55:04,691 DEBUG [Thread-1542] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:04,740 DEBUG [Thread-1538] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:60465 2023-04-01 14:55:04,740 DEBUG [Thread-1538] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 20 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 163 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 181 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4804 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4711 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4535 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4825 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4725 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 14:55:04,741 INFO [Listener at localhost/46667] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 14:55:04,741 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:60465 2023-04-01 14:55:04,741 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:04,742 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 14:55:04,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-01 14:55:04,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:04,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 14:55:04,745 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360904745"}]},"ts":"1680360904745"} 2023-04-01 14:55:04,746 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 14:55:04,747 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 14:55:04,748 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5326183795415453ea78c16f1f182c7d, UNASSIGN}] 2023-04-01 14:55:04,749 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5326183795415453ea78c16f1f182c7d, UNASSIGN 2023-04-01 14:55:04,749 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=5326183795415453ea78c16f1f182c7d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:04,750 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360904749"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360904749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360904749"}]},"ts":"1680360904749"} 2023-04-01 14:55:04,750 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:55:04,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 14:55:04,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5326183795415453ea78c16f1f182c7d 2023-04-01 14:55:04,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5326183795415453ea78c16f1f182c7d, disabling compactions & flushes 2023-04-01 14:55:04,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:55:04,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:55:04,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. after waiting 0 ms 2023-04-01 14:55:04,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:55:04,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5326183795415453ea78c16f1f182c7d 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-01 14:55:04,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=A 2023-04-01 14:55:04,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:04,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=B 2023-04-01 14:55:04,903 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:04,903 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5326183795415453ea78c16f1f182c7d, store=C 2023-04-01 14:55:04,903 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:05,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 14:55:05,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/104a09d4cd2f428ba101e0713ac45b2d 2023-04-01 14:55:05,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/97558d65b9b149a3b3925029316bf1be 2023-04-01 14:55:05,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 14:55:05,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/f45f0f43e29a4d3ca7c463a0bd5d0b4a 2023-04-01 14:55:05,736 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/A/104a09d4cd2f428ba101e0713ac45b2d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/104a09d4cd2f428ba101e0713ac45b2d 2023-04-01 14:55:05,742 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/104a09d4cd2f428ba101e0713ac45b2d, entries=150, sequenceid=514, filesize=11.9 K 2023-04-01 14:55:05,742 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/B/97558d65b9b149a3b3925029316bf1be as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/97558d65b9b149a3b3925029316bf1be 2023-04-01 14:55:05,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/97558d65b9b149a3b3925029316bf1be, entries=150, sequenceid=514, filesize=11.9 K 2023-04-01 14:55:05,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/.tmp/C/f45f0f43e29a4d3ca7c463a0bd5d0b4a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f45f0f43e29a4d3ca7c463a0bd5d0b4a 2023-04-01 14:55:05,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f45f0f43e29a4d3ca7c463a0bd5d0b4a, entries=150, sequenceid=514, filesize=11.9 K 2023-04-01 14:55:05,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 5326183795415453ea78c16f1f182c7d in 856ms, sequenceid=514, compaction requested=false 2023-04-01 14:55:05,785 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/383b832b887843748d036532638ced0e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/c6fe2920e96645dea42dfcb826a519b6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/82f8942348f74ed6b93382fe339257e2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e62a8e99e3674f7390a1021a8dc8b7b8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/25fdd2bdc34c4734af9886077274bbf4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/0d06158a6a6c4dc1aa4889cdea14f444, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/41df6e8cbe414fd9bad29a9f2d29aea6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d527fedf69504d13b3189700daad15de, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/3ae75540f92a4a228df4f493edff6d2c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/f300dddd81bc4baa81ee3b7c60275d1f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/a67a308f5d9344bbb99f07bafe7d8b50, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95e74d6a51e64c87a9d5bc5836cc7428, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d4c8b111a6324586b8743cccfcccb648, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/1d5b2eef22034e96bd762f9104ec25bb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/9c5b3bb9fa8344799e41dde0e7c51255, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/79a5be2ce6734d67bda232a14038761b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/69c790d897f94ae9be33e86ab9c115d4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e6773508d0d44712af6555cb59476424, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/b9ab9045ccca499a9cb288e61cd301b6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e4edf08d52cf4286a8d380c797fbb989, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/543dc2ffbc8a49fb9dd0afe750fb3bbe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/31a572fd817147079384218dd79b41f8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/21e03570caca4506892da30d4a02795a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/19ce98178aa040e2a11745c80fb44522, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/6eb09658f0bc46fca68788c1fe5c9ef4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/25b2dbb373ce4e54809494bdcb4d7fe1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/cc74b1a47f6f4435b41d958aa34b578b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/bbd8d2d27731469cbe944f452a75aacf, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95c307e89e014c87b719e6cd1a157167, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/28983e6b5fd24518b5e8987b7246c82a] to archive 2023-04-01 14:55:05,786 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:55:05,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/383b832b887843748d036532638ced0e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/383b832b887843748d036532638ced0e 2023-04-01 14:55:05,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/c6fe2920e96645dea42dfcb826a519b6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/c6fe2920e96645dea42dfcb826a519b6 2023-04-01 14:55:05,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/82f8942348f74ed6b93382fe339257e2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/82f8942348f74ed6b93382fe339257e2 2023-04-01 14:55:05,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e62a8e99e3674f7390a1021a8dc8b7b8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e62a8e99e3674f7390a1021a8dc8b7b8 2023-04-01 14:55:05,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/25fdd2bdc34c4734af9886077274bbf4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/25fdd2bdc34c4734af9886077274bbf4 2023-04-01 14:55:05,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/0d06158a6a6c4dc1aa4889cdea14f444 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/0d06158a6a6c4dc1aa4889cdea14f444 2023-04-01 14:55:05,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/41df6e8cbe414fd9bad29a9f2d29aea6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/41df6e8cbe414fd9bad29a9f2d29aea6 2023-04-01 14:55:05,799 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d527fedf69504d13b3189700daad15de to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d527fedf69504d13b3189700daad15de 2023-04-01 14:55:05,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/3ae75540f92a4a228df4f493edff6d2c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/3ae75540f92a4a228df4f493edff6d2c 2023-04-01 14:55:05,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/f300dddd81bc4baa81ee3b7c60275d1f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/f300dddd81bc4baa81ee3b7c60275d1f 2023-04-01 14:55:05,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/a67a308f5d9344bbb99f07bafe7d8b50 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/a67a308f5d9344bbb99f07bafe7d8b50 2023-04-01 14:55:05,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95e74d6a51e64c87a9d5bc5836cc7428 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95e74d6a51e64c87a9d5bc5836cc7428 2023-04-01 14:55:05,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d4c8b111a6324586b8743cccfcccb648 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/d4c8b111a6324586b8743cccfcccb648 2023-04-01 14:55:05,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/1d5b2eef22034e96bd762f9104ec25bb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/1d5b2eef22034e96bd762f9104ec25bb 2023-04-01 14:55:05,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/9c5b3bb9fa8344799e41dde0e7c51255 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/9c5b3bb9fa8344799e41dde0e7c51255 2023-04-01 14:55:05,808 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/79a5be2ce6734d67bda232a14038761b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/79a5be2ce6734d67bda232a14038761b 2023-04-01 14:55:05,809 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/69c790d897f94ae9be33e86ab9c115d4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/69c790d897f94ae9be33e86ab9c115d4 2023-04-01 14:55:05,811 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e6773508d0d44712af6555cb59476424 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e6773508d0d44712af6555cb59476424 2023-04-01 14:55:05,812 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/b9ab9045ccca499a9cb288e61cd301b6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/b9ab9045ccca499a9cb288e61cd301b6 2023-04-01 14:55:05,814 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e4edf08d52cf4286a8d380c797fbb989 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/e4edf08d52cf4286a8d380c797fbb989 2023-04-01 14:55:05,815 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/543dc2ffbc8a49fb9dd0afe750fb3bbe to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/543dc2ffbc8a49fb9dd0afe750fb3bbe 2023-04-01 14:55:05,816 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/31a572fd817147079384218dd79b41f8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/31a572fd817147079384218dd79b41f8 2023-04-01 14:55:05,817 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/21e03570caca4506892da30d4a02795a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/21e03570caca4506892da30d4a02795a 2023-04-01 14:55:05,818 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/19ce98178aa040e2a11745c80fb44522 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/19ce98178aa040e2a11745c80fb44522 2023-04-01 14:55:05,819 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/6eb09658f0bc46fca68788c1fe5c9ef4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/6eb09658f0bc46fca68788c1fe5c9ef4 2023-04-01 14:55:05,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/25b2dbb373ce4e54809494bdcb4d7fe1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/25b2dbb373ce4e54809494bdcb4d7fe1 2023-04-01 14:55:05,821 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/cc74b1a47f6f4435b41d958aa34b578b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/cc74b1a47f6f4435b41d958aa34b578b 2023-04-01 14:55:05,822 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/bbd8d2d27731469cbe944f452a75aacf to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/bbd8d2d27731469cbe944f452a75aacf 2023-04-01 14:55:05,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95c307e89e014c87b719e6cd1a157167 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/95c307e89e014c87b719e6cd1a157167 2023-04-01 14:55:05,825 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/28983e6b5fd24518b5e8987b7246c82a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/28983e6b5fd24518b5e8987b7246c82a 2023-04-01 14:55:05,833 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/6b2823765b274968917ed174e52b48fe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/9c2406f728a24738849762a1f18f05de, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/338af4d480bb4ae1bb2f8f186ce782e3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/4b6e5e79fe3f45c18249d843b2118a48, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/798e210c714e41e68f7382a567dc0f6c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/ad54c2b2667d432e9f16ac65545951dc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/602022782c3b4e46b64535e23107acd5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/76628ee139fa4a29818723dc619e5d2c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/bac363b605ab48be92a67279cbccd37b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/806f135bd38f4f4ab828f145db7b7407, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cedd6a2ff8c4527b9c48b4ddc185b95, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/f17fa2a3cb424b5083aa7637e4f549a9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3dbb8533546a4063b01fe0006a55d462, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/cc666f9ad04448beb331868bd6933dab, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/94c0ea64281346729d4d02403741ec6f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/8de146800bb449b78dcae7824c2b9190, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1a16b22ed4ba4e9986d99404b39d9bac, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/063813620c0c463b8928ada40fe87294, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/51909c0a9c84442dad9d50875fc6f714, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/e0a13db1df324174a1e3c424dc24d498, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/51d5656887764e5fb4f78bb0c4bd9a04, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/4531393e38e84cdbbe834c88452c01bc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/da20177158954b56884abc0bd8b1286e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/97605c4ca67a4ef486c32789d0e03013, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/7f94dac7e5634375acf582ee819f1e86, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/fcf2fefeaf694f2abd5224ecc124fa07, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/a414cd37d5ff4257b8b4c4c2a0b63827, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/71c3067cae584b4aaaa0713b07f9935f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1b5ec99d283d492580dd6b48a29366c4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cb7a15f79e6444a8d8097aff7698346] to archive 2023-04-01 14:55:05,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:55:05,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/6b2823765b274968917ed174e52b48fe to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/6b2823765b274968917ed174e52b48fe 2023-04-01 14:55:05,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/9c2406f728a24738849762a1f18f05de to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/9c2406f728a24738849762a1f18f05de 2023-04-01 14:55:05,838 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/338af4d480bb4ae1bb2f8f186ce782e3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/338af4d480bb4ae1bb2f8f186ce782e3 2023-04-01 14:55:05,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/4b6e5e79fe3f45c18249d843b2118a48 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/4b6e5e79fe3f45c18249d843b2118a48 2023-04-01 14:55:05,840 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/798e210c714e41e68f7382a567dc0f6c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/798e210c714e41e68f7382a567dc0f6c 2023-04-01 14:55:05,841 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/ad54c2b2667d432e9f16ac65545951dc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/ad54c2b2667d432e9f16ac65545951dc 2023-04-01 14:55:05,842 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/602022782c3b4e46b64535e23107acd5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/602022782c3b4e46b64535e23107acd5 2023-04-01 14:55:05,843 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/76628ee139fa4a29818723dc619e5d2c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/76628ee139fa4a29818723dc619e5d2c 2023-04-01 14:55:05,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/bac363b605ab48be92a67279cbccd37b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/bac363b605ab48be92a67279cbccd37b 2023-04-01 14:55:05,846 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/806f135bd38f4f4ab828f145db7b7407 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/806f135bd38f4f4ab828f145db7b7407 2023-04-01 14:55:05,847 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cedd6a2ff8c4527b9c48b4ddc185b95 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cedd6a2ff8c4527b9c48b4ddc185b95 2023-04-01 14:55:05,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 14:55:05,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/f17fa2a3cb424b5083aa7637e4f549a9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/f17fa2a3cb424b5083aa7637e4f549a9 2023-04-01 14:55:05,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3dbb8533546a4063b01fe0006a55d462 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3dbb8533546a4063b01fe0006a55d462 2023-04-01 14:55:05,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/cc666f9ad04448beb331868bd6933dab to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/cc666f9ad04448beb331868bd6933dab 2023-04-01 14:55:05,853 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/94c0ea64281346729d4d02403741ec6f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/94c0ea64281346729d4d02403741ec6f 2023-04-01 14:55:05,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/8de146800bb449b78dcae7824c2b9190 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/8de146800bb449b78dcae7824c2b9190 2023-04-01 14:55:05,855 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1a16b22ed4ba4e9986d99404b39d9bac to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1a16b22ed4ba4e9986d99404b39d9bac 2023-04-01 14:55:05,856 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/063813620c0c463b8928ada40fe87294 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/063813620c0c463b8928ada40fe87294 2023-04-01 14:55:05,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/51909c0a9c84442dad9d50875fc6f714 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/51909c0a9c84442dad9d50875fc6f714 2023-04-01 14:55:05,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/e0a13db1df324174a1e3c424dc24d498 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/e0a13db1df324174a1e3c424dc24d498 2023-04-01 14:55:05,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/51d5656887764e5fb4f78bb0c4bd9a04 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/51d5656887764e5fb4f78bb0c4bd9a04 2023-04-01 14:55:05,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/4531393e38e84cdbbe834c88452c01bc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/4531393e38e84cdbbe834c88452c01bc 2023-04-01 14:55:05,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/da20177158954b56884abc0bd8b1286e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/da20177158954b56884abc0bd8b1286e 2023-04-01 14:55:05,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/97605c4ca67a4ef486c32789d0e03013 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/97605c4ca67a4ef486c32789d0e03013 2023-04-01 14:55:05,863 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/7f94dac7e5634375acf582ee819f1e86 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/7f94dac7e5634375acf582ee819f1e86 2023-04-01 14:55:05,864 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/fcf2fefeaf694f2abd5224ecc124fa07 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/fcf2fefeaf694f2abd5224ecc124fa07 2023-04-01 14:55:05,865 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/a414cd37d5ff4257b8b4c4c2a0b63827 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/a414cd37d5ff4257b8b4c4c2a0b63827 2023-04-01 14:55:05,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/71c3067cae584b4aaaa0713b07f9935f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/71c3067cae584b4aaaa0713b07f9935f 2023-04-01 14:55:05,867 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1b5ec99d283d492580dd6b48a29366c4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/1b5ec99d283d492580dd6b48a29366c4 2023-04-01 14:55:05,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cb7a15f79e6444a8d8097aff7698346 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/3cb7a15f79e6444a8d8097aff7698346 2023-04-01 14:55:05,870 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4dea448c9ec349118dd5aa20b02d98d4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/785a0bea67ea478fa68341376020c2b0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/539ea0f4c71042bcb5dda63d237faa29, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/b3bc17bd55f84da5af1198debd8081ce, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e443e1f430af4b7abb623b21a6741327, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/04cbcefa475b475ba724f7b54f396737, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e2a1b28137ed424fb7cda39b73752be9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/825a09aeaade48f580b3ee46a4dad93c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/926775ebef074768a01f9238d3132ed1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/5763084e80664913b007203107214c95, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/2f28591410814215b2c28f01e5edfbbe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fde4113321c44c2797ee94573a67c79a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/c9848e121bbc4dee981131070de1f14f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e99527f73a894d42a124a915a7a3a406, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a06599a032cb49888f837b3ae18cc466, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a959a00641284f90b9024d10fbafd7dd, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f9b5982cae604683809b82fc4f34f0b5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4375f41e92c045cfbb273c9db766b1d9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f99c658e03b14f76afc4ee812dd7cb92, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f20964d95b6a472fa600226c2bda83db, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/05deecb45af24fc296bbc4e08c01da02, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/63d7992cfedb4b01873ab44d8f445dc0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4ebc1f3630774cedb14aad6ad7e938bc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/29f6d375ebd34270802351cbda637a37, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fee7b90339904eb08790c9051cc51e22, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/6b45d751cda2470687ac7a5ae0c36f9e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/c87f5bb30299474a8dfb0e8b670a70a0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4e1ee337dd5c4d108769ed97be658192, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/0b94ed70ce254228a26cb989ada08014, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/790cb8c58acb4a5face0eb1f0866da34] to archive 2023-04-01 14:55:05,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:55:05,872 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4dea448c9ec349118dd5aa20b02d98d4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4dea448c9ec349118dd5aa20b02d98d4 2023-04-01 14:55:05,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/785a0bea67ea478fa68341376020c2b0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/785a0bea67ea478fa68341376020c2b0 2023-04-01 14:55:05,874 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/539ea0f4c71042bcb5dda63d237faa29 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/539ea0f4c71042bcb5dda63d237faa29 2023-04-01 14:55:05,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/b3bc17bd55f84da5af1198debd8081ce to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/b3bc17bd55f84da5af1198debd8081ce 2023-04-01 14:55:05,876 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e443e1f430af4b7abb623b21a6741327 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e443e1f430af4b7abb623b21a6741327 2023-04-01 14:55:05,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/04cbcefa475b475ba724f7b54f396737 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/04cbcefa475b475ba724f7b54f396737 2023-04-01 14:55:05,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e2a1b28137ed424fb7cda39b73752be9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e2a1b28137ed424fb7cda39b73752be9 2023-04-01 14:55:05,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/825a09aeaade48f580b3ee46a4dad93c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/825a09aeaade48f580b3ee46a4dad93c 2023-04-01 14:55:05,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/926775ebef074768a01f9238d3132ed1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/926775ebef074768a01f9238d3132ed1 2023-04-01 14:55:05,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/5763084e80664913b007203107214c95 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/5763084e80664913b007203107214c95 2023-04-01 14:55:05,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/2f28591410814215b2c28f01e5edfbbe to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/2f28591410814215b2c28f01e5edfbbe 2023-04-01 14:55:05,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fde4113321c44c2797ee94573a67c79a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fde4113321c44c2797ee94573a67c79a 2023-04-01 14:55:05,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/c9848e121bbc4dee981131070de1f14f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/c9848e121bbc4dee981131070de1f14f 2023-04-01 14:55:05,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e99527f73a894d42a124a915a7a3a406 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/e99527f73a894d42a124a915a7a3a406 2023-04-01 14:55:05,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a06599a032cb49888f837b3ae18cc466 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a06599a032cb49888f837b3ae18cc466 2023-04-01 14:55:05,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a959a00641284f90b9024d10fbafd7dd to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/a959a00641284f90b9024d10fbafd7dd 2023-04-01 14:55:05,888 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f9b5982cae604683809b82fc4f34f0b5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f9b5982cae604683809b82fc4f34f0b5 2023-04-01 14:55:05,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4375f41e92c045cfbb273c9db766b1d9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4375f41e92c045cfbb273c9db766b1d9 2023-04-01 14:55:05,890 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f99c658e03b14f76afc4ee812dd7cb92 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f99c658e03b14f76afc4ee812dd7cb92 2023-04-01 14:55:05,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f20964d95b6a472fa600226c2bda83db to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f20964d95b6a472fa600226c2bda83db 2023-04-01 14:55:05,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/05deecb45af24fc296bbc4e08c01da02 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/05deecb45af24fc296bbc4e08c01da02 2023-04-01 14:55:05,892 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/63d7992cfedb4b01873ab44d8f445dc0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/63d7992cfedb4b01873ab44d8f445dc0 2023-04-01 14:55:05,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4ebc1f3630774cedb14aad6ad7e938bc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4ebc1f3630774cedb14aad6ad7e938bc 2023-04-01 14:55:05,894 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/29f6d375ebd34270802351cbda637a37 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/29f6d375ebd34270802351cbda637a37 2023-04-01 14:55:05,895 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fee7b90339904eb08790c9051cc51e22 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/fee7b90339904eb08790c9051cc51e22 2023-04-01 14:55:05,896 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/6b45d751cda2470687ac7a5ae0c36f9e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/6b45d751cda2470687ac7a5ae0c36f9e 2023-04-01 14:55:05,897 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/c87f5bb30299474a8dfb0e8b670a70a0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/c87f5bb30299474a8dfb0e8b670a70a0 2023-04-01 14:55:05,898 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4e1ee337dd5c4d108769ed97be658192 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/4e1ee337dd5c4d108769ed97be658192 2023-04-01 14:55:05,899 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/0b94ed70ce254228a26cb989ada08014 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/0b94ed70ce254228a26cb989ada08014 2023-04-01 14:55:05,899 DEBUG [StoreCloser-TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/790cb8c58acb4a5face0eb1f0866da34 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/790cb8c58acb4a5face0eb1f0866da34 2023-04-01 14:55:05,903 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/recovered.edits/517.seqid, newMaxSeqId=517, maxSeqId=1 2023-04-01 14:55:05,904 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d. 2023-04-01 14:55:05,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5326183795415453ea78c16f1f182c7d: 2023-04-01 14:55:05,906 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5326183795415453ea78c16f1f182c7d 2023-04-01 14:55:05,906 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=5326183795415453ea78c16f1f182c7d, regionState=CLOSED 2023-04-01 14:55:05,906 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360905906"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360905906"}]},"ts":"1680360905906"} 2023-04-01 14:55:05,908 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-01 14:55:05,909 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 5326183795415453ea78c16f1f182c7d, server=jenkins-hbase4.apache.org,40863,1680360818567 in 1.1570 sec 2023-04-01 14:55:05,909 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-01 14:55:05,910 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5326183795415453ea78c16f1f182c7d, UNASSIGN in 1.1600 sec 2023-04-01 14:55:05,910 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360905910"}]},"ts":"1680360905910"} 2023-04-01 14:55:05,911 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 14:55:05,912 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 14:55:05,914 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1710 sec 2023-04-01 14:55:06,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-01 14:55:06,849 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-01 14:55:06,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-01 14:55:06,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:06,852 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:06,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-01 14:55:06,852 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:06,855 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d 2023-04-01 14:55:06,857 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/recovered.edits] 2023-04-01 14:55:06,859 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/104a09d4cd2f428ba101e0713ac45b2d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/104a09d4cd2f428ba101e0713ac45b2d 2023-04-01 14:55:06,860 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/156efed779b64b72bcc62d768eb371ff to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/A/156efed779b64b72bcc62d768eb371ff 2023-04-01 14:55:06,862 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/97558d65b9b149a3b3925029316bf1be to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/97558d65b9b149a3b3925029316bf1be 2023-04-01 14:55:06,863 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/f19b97ce339547b59641aa9a6d8b4b8e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/B/f19b97ce339547b59641aa9a6d8b4b8e 2023-04-01 14:55:06,865 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/780665ef3c6b427b966fb99f64f913a1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/780665ef3c6b427b966fb99f64f913a1 2023-04-01 14:55:06,866 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f45f0f43e29a4d3ca7c463a0bd5d0b4a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/C/f45f0f43e29a4d3ca7c463a0bd5d0b4a 2023-04-01 14:55:06,869 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/recovered.edits/517.seqid to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d/recovered.edits/517.seqid 2023-04-01 14:55:06,869 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/5326183795415453ea78c16f1f182c7d 2023-04-01 14:55:06,869 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:55:06,871 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:06,872 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 14:55:06,874 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 14:55:06,874 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:06,874 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 14:55:06,875 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680360906874"}]},"ts":"9223372036854775807"} 2023-04-01 14:55:06,876 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 14:55:06,876 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5326183795415453ea78c16f1f182c7d, NAME => 'TestAcidGuarantees,,1680360873972.5326183795415453ea78c16f1f182c7d.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 14:55:06,876 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 14:55:06,876 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680360906876"}]},"ts":"9223372036854775807"} 2023-04-01 14:55:06,877 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 14:55:06,879 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:06,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 29 msec 2023-04-01 14:55:06,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-01 14:55:06,953 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-01 14:55:06,964 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 239), OpenFileDescriptor=521 (was 552), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=363 (was 395), ProcessCount=169 (was 169), AvailableMemoryMB=3279 (was 3342) 2023-04-01 14:55:06,975 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=521, MaxFileDescriptor=60000, SystemLoadAverage=363, ProcessCount=169, AvailableMemoryMB=3278 2023-04-01 14:55:06,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 14:55:06,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 14:55:06,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:06,979 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 14:55:06,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-01 14:55:06,980 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 14:55:06,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 14:55:06,981 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:06,981 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47 empty. 2023-04-01 14:55:06,981 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:06,982 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:55:07,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 14:55:07,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 14:55:07,388 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 14:55:07,390 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => b2dc695f920ae42c5964c2b96bd0ec47, NAME => 'TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp 2023-04-01 14:55:07,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 14:55:07,806 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:55:07,806 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing b2dc695f920ae42c5964c2b96bd0ec47, disabling compactions & flushes 2023-04-01 14:55:07,806 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:07,806 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:07,806 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. after waiting 0 ms 2023-04-01 14:55:07,806 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:07,806 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:07,806 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:07,808 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 14:55:07,809 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360907809"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360907809"}]},"ts":"1680360907809"} 2023-04-01 14:55:07,810 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 14:55:07,810 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 14:55:07,811 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360907810"}]},"ts":"1680360907810"} 2023-04-01 14:55:07,811 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 14:55:07,815 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b2dc695f920ae42c5964c2b96bd0ec47, ASSIGN}] 2023-04-01 14:55:07,816 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b2dc695f920ae42c5964c2b96bd0ec47, ASSIGN 2023-04-01 14:55:07,817 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b2dc695f920ae42c5964c2b96bd0ec47, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40863,1680360818567; forceNewPlan=false, retain=false 2023-04-01 14:55:07,968 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=b2dc695f920ae42c5964c2b96bd0ec47, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:07,968 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360907967"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360907967"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360907967"}]},"ts":"1680360907967"} 2023-04-01 14:55:07,969 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:55:08,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 14:55:08,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:08,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b2dc695f920ae42c5964c2b96bd0ec47, NAME => 'TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:55:08,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:08,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:55:08,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:08,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:08,125 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:08,126 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A 2023-04-01 14:55:08,126 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A 2023-04-01 14:55:08,126 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:55:08,126 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b2dc695f920ae42c5964c2b96bd0ec47 columnFamilyName A 2023-04-01 14:55:08,127 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(310): Store=b2dc695f920ae42c5964c2b96bd0ec47/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:55:08,127 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:08,128 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B 2023-04-01 14:55:08,128 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B 2023-04-01 14:55:08,128 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:55:08,128 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b2dc695f920ae42c5964c2b96bd0ec47 columnFamilyName B 2023-04-01 14:55:08,129 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(310): Store=b2dc695f920ae42c5964c2b96bd0ec47/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:55:08,129 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:08,130 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C 2023-04-01 14:55:08,130 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C 2023-04-01 14:55:08,130 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:55:08,130 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b2dc695f920ae42c5964c2b96bd0ec47 columnFamilyName C 2023-04-01 14:55:08,130 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(310): Store=b2dc695f920ae42c5964c2b96bd0ec47/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:55:08,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:08,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:08,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:08,133 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 14:55:08,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:08,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 14:55:08,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b2dc695f920ae42c5964c2b96bd0ec47; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72326951, jitterRate=0.07775555551052094}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 14:55:08,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:08,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., pid=37, masterSystemTime=1680360908120 2023-04-01 14:55:08,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:08,138 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:08,138 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=b2dc695f920ae42c5964c2b96bd0ec47, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:08,138 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360908138"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680360908138"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680360908138"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680360908138"}]},"ts":"1680360908138"} 2023-04-01 14:55:08,140 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-01 14:55:08,140 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 in 170 msec 2023-04-01 14:55:08,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-01 14:55:08,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b2dc695f920ae42c5964c2b96bd0ec47, ASSIGN in 325 msec 2023-04-01 14:55:08,142 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 14:55:08,142 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360908142"}]},"ts":"1680360908142"} 2023-04-01 14:55:08,143 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 14:55:08,144 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 14:55:08,145 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1670 sec 2023-04-01 14:55:09,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-01 14:55:09,084 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-01 14:55:09,085 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x1a5ab074 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:09,090 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bb27fd4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:09,091 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:09,092 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43464, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:09,093 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 14:55:09,094 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 14:55:09,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 14:55:09,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 14:55:09,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:09,507 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 14:55:09,508 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 14:55:09,508 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 14:55:09,510 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-01 14:55:09,513 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b2dc695f920ae42c5964c2b96bd0ec47, REOPEN/MOVE}] 2023-04-01 14:55:09,513 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b2dc695f920ae42c5964c2b96bd0ec47, REOPEN/MOVE 2023-04-01 14:55:09,514 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b2dc695f920ae42c5964c2b96bd0ec47, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:09,514 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360909514"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360909514"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360909514"}]},"ts":"1680360909514"} 2023-04-01 14:55:09,515 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:55:09,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:09,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b2dc695f920ae42c5964c2b96bd0ec47, disabling compactions & flushes 2023-04-01 14:55:09,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:09,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:09,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. after waiting 0 ms 2023-04-01 14:55:09,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:09,671 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-01 14:55:09,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:09,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:09,672 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: b2dc695f920ae42c5964c2b96bd0ec47 to self. 2023-04-01 14:55:09,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:09,674 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b2dc695f920ae42c5964c2b96bd0ec47, regionState=CLOSED 2023-04-01 14:55:09,674 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360909674"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360909674"}]},"ts":"1680360909674"} 2023-04-01 14:55:09,676 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-01 14:55:09,676 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 in 160 msec 2023-04-01 14:55:09,676 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b2dc695f920ae42c5964c2b96bd0ec47, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40863,1680360818567; forceNewPlan=false, retain=true 2023-04-01 14:55:09,827 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b2dc695f920ae42c5964c2b96bd0ec47, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:09,827 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360909827"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360909827"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360909827"}]},"ts":"1680360909827"} 2023-04-01 14:55:09,828 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:55:09,983 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:09,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b2dc695f920ae42c5964c2b96bd0ec47, NAME => 'TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:55:09,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:09,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:55:09,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:09,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:09,985 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:09,985 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A 2023-04-01 14:55:09,985 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A 2023-04-01 14:55:09,986 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:55:09,986 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b2dc695f920ae42c5964c2b96bd0ec47 columnFamilyName A 2023-04-01 14:55:09,987 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(310): Store=b2dc695f920ae42c5964c2b96bd0ec47/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:55:09,988 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:09,989 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B 2023-04-01 14:55:09,989 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B 2023-04-01 14:55:09,989 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:55:09,989 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b2dc695f920ae42c5964c2b96bd0ec47 columnFamilyName B 2023-04-01 14:55:09,990 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(310): Store=b2dc695f920ae42c5964c2b96bd0ec47/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:55:09,990 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:09,990 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C 2023-04-01 14:55:09,990 DEBUG [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C 2023-04-01 14:55:09,991 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:55:09,991 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b2dc695f920ae42c5964c2b96bd0ec47 columnFamilyName C 2023-04-01 14:55:09,991 INFO [StoreOpener-b2dc695f920ae42c5964c2b96bd0ec47-1] regionserver.HStore(310): Store=b2dc695f920ae42c5964c2b96bd0ec47/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:55:09,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:09,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:09,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:09,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 14:55:09,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:09,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b2dc695f920ae42c5964c2b96bd0ec47; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62365424, jitterRate=-0.07068276405334473}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 14:55:09,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:09,996 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., pid=42, masterSystemTime=1680360909980 2023-04-01 14:55:09,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:09,997 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:09,998 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b2dc695f920ae42c5964c2b96bd0ec47, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:09,998 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360909998"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680360909998"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680360909998"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680360909998"}]},"ts":"1680360909998"} 2023-04-01 14:55:09,999 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-01 14:55:09,999 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 in 171 msec 2023-04-01 14:55:10,000 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-01 14:55:10,000 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b2dc695f920ae42c5964c2b96bd0ec47, REOPEN/MOVE in 486 msec 2023-04-01 14:55:10,002 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-01 14:55:10,002 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 491 msec 2023-04-01 14:55:10,003 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-04-01 14:55:10,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-01 14:55:10,004 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x0bcfc1e0 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:10,008 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@445919fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:10,009 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:10,012 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:10,013 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:10,017 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:10,017 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:10,021 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:10,021 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:10,024 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:10,024 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x51ccd8b7 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:10,028 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21128196, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:10,028 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x68a0e21a to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:10,033 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e86d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:10,033 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:10,036 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:10,036 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:10,039 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:10,040 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x3e079da1 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:10,042 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eae4191, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:10,045 DEBUG [hconnection-0x504a11a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:10,046 DEBUG [hconnection-0x36ebdfe4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:10,046 DEBUG [hconnection-0x65599ce2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:10,049 DEBUG [hconnection-0x6cad0de6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:10,050 DEBUG [hconnection-0x30797067-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:10,050 DEBUG [hconnection-0xdefc48a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:10,050 DEBUG [hconnection-0x5504ba0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:10,051 DEBUG [hconnection-0x2f204606-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:10,051 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43480, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:10,051 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:10,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:10,052 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43466, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:10,052 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43492, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:10,052 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43524, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:10,053 DEBUG [hconnection-0x5f50944a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:10,054 DEBUG [hconnection-0x1f85c2da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:10,055 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43514, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:10,057 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43544, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:10,061 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:10,061 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43534, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:10,062 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43522, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:10,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:10,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:10,065 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:10,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:10,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:10,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:10,067 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,067 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:10,070 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:55:10,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:10,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:10,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:10,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:10,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:10,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:10,073 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,074 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:10,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:10,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:10,074 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,075 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:10,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:10,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:10,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:10,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:10,084 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:10,084 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:10,084 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:10,085 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:10,085 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:10,085 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:10,085 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:10,085 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:10,085 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:10,085 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:10,085 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:10,085 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,088 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:10,088 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:10,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,090 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:10,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1680360970086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:10,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:10,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:10,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b50a082[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:10,092 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:10,092 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360970088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360970089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360970090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360970092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,097 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,097 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,097 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:10,097 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:10,097 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:10,098 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:10,099 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,099 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:10,099 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:10,100 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:10,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:10,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:10,102 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:10,102 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:10,102 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:10,102 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:10,102 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:10,104 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:10,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:10,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:10,105 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,107 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,108 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,108 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:10,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:10,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,109 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401cdd08173719f499b9c8c4d73ac2a2fc9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401cdd08173719f499b9c8c4d73ac2a2fc9 2023-04-01 14:55:10,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,113 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,113 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,113 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,113 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:10,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:10,113 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:10,113 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:10,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:10,113 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,114 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,114 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:10,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:10,113 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:10,114 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:10,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:10,115 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:10,129 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/dbf51ab2b2984b28b3227da93b3da425 2023-04-01 14:55:10,164 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/efedb47336ff4e20ad763f998fb9b24b 2023-04-01 14:55:10,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/bf202651dc994e73b2ab8f84b1976654 2023-04-01 14:55:10,192 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/dbf51ab2b2984b28b3227da93b3da425 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/dbf51ab2b2984b28b3227da93b3da425 2023-04-01 14:55:10,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1680360970191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/dbf51ab2b2984b28b3227da93b3da425, entries=250, sequenceid=17, filesize=38.7 K 2023-04-01 14:55:10,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360970196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/efedb47336ff4e20ad763f998fb9b24b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/efedb47336ff4e20ad763f998fb9b24b 2023-04-01 14:55:10,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360970196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360970196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360970196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/efedb47336ff4e20ad763f998fb9b24b, entries=150, sequenceid=17, filesize=11.6 K 2023-04-01 14:55:10,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/bf202651dc994e73b2ab8f84b1976654 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bf202651dc994e73b2ab8f84b1976654 2023-04-01 14:55:10,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bf202651dc994e73b2ab8f84b1976654, entries=150, sequenceid=17, filesize=11.6 K 2023-04-01 14:55:10,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for b2dc695f920ae42c5964c2b96bd0ec47 in 134ms, sequenceid=17, compaction requested=false 2023-04-01 14:55:10,205 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-01 14:55:10,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:10,215 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:10,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:10,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:10,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:10,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:10,218 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:10,218 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:10,218 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:10,218 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:10,218 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,220 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:10,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:10,220 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,220 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,220 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:10,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:10,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:10,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:10,224 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:10,227 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:10,228 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:10,228 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:10,228 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:10,228 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 14:55:10,228 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:10,228 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:10,228 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:10,228 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:10,228 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:10,228 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:10,243 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401fe8bd207ec0b4bcf8fa0975a7faa667a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401fe8bd207ec0b4bcf8fa0975a7faa667a 2023-04-01 14:55:10,253 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/0abc269841ad4779a645e6a4b47a8f9c 2023-04-01 14:55:10,288 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/a6b6c88c5307458bb8894230c256ffea 2023-04-01 14:55:10,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:10,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:10,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360970402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360970403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360970405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1680360970405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360970406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360970507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360970507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360970510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1680360970510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360970511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,699 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/9175178b572b4718baa688e694332d48 2023-04-01 14:55:10,704 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/0abc269841ad4779a645e6a4b47a8f9c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/0abc269841ad4779a645e6a4b47a8f9c 2023-04-01 14:55:10,707 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/0abc269841ad4779a645e6a4b47a8f9c, entries=150, sequenceid=37, filesize=25.1 K 2023-04-01 14:55:10,708 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/a6b6c88c5307458bb8894230c256ffea as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a6b6c88c5307458bb8894230c256ffea 2023-04-01 14:55:10,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360970710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360970710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360970715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1680360970715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,717 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a6b6c88c5307458bb8894230c256ffea, entries=150, sequenceid=37, filesize=11.6 K 2023-04-01 14:55:10,718 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/9175178b572b4718baa688e694332d48 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9175178b572b4718baa688e694332d48 2023-04-01 14:55:10,722 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9175178b572b4718baa688e694332d48, entries=150, sequenceid=37, filesize=11.6 K 2023-04-01 14:55:10,722 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for b2dc695f920ae42c5964c2b96bd0ec47 in 494ms, sequenceid=37, compaction requested=false 2023-04-01 14:55:10,724 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:10,724 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:10,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:10,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:10,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:10,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:10,727 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,727 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:10,727 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,728 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:10,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:10,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@131011de[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:10,729 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:10,729 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,730 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,730 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:10,730 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:10,730 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:10,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:10,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:10,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:10,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:10,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:10,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:10,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:10,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:10,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:10,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:10,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:10,735 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:10,735 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:10,735 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:10,735 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:10,736 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:10,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:10,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:10,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:10,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:10,739 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,744 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,744 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:10,744 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:10,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:10,744 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:10,744 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:10,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:10,744 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:10,744 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,744 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,744 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:10,744 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:10,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:10,744 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,745 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360970821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,845 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:10,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:10,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:10,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:10,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:10,864 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:10,864 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:10,864 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:10,864 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:10,865 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,865 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,866 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,866 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:10,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:10,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:10,867 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,867 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:10,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:10,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:10,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:10,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:10,868 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:10,868 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:10,868 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:10,869 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:10,869 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:10,869 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:10,869 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:10,869 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:10,869 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:10,869 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:10,869 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:10,869 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,869 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:10,869 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:10,870 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,870 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:10,871 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,871 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:10,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:10,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@398d2b65[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:10,873 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:10,873 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,874 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:10,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:10,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:10,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,875 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:10,877 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:10,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:10,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:10,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:10,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:10,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:10,880 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:10,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:10,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:10,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:10,880 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:10,881 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,881 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,881 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:10,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:10,882 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,883 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:10,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:10,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:10,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:10,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:10,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:10,887 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:10,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:10,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,888 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:10,888 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:10,888 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:10,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:10,888 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:10,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:10,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:10,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360970927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:10,988 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:10,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:10,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:11,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:11,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:11,000 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:11,000 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,001 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:11,001 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:11,001 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,001 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,002 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,002 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:11,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:11,003 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,003 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:11,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:11,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:11,007 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,008 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:11,008 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:11,008 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,008 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:11,008 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:11,008 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,008 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:11,008 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:11,010 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,011 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,015 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,015 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:11,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30dd534b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:11,017 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:11,018 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,019 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,019 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,019 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:11,019 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:11,019 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:11,019 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:11,023 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:11,023 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:11,023 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:11,023 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:11,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360971017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:11,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1680360971018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:11,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360971020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:11,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:11,028 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:11,028 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:11,028 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:11,028 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:11,028 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:11,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:11,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360971023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,030 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,031 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,031 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,031 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:11,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,032 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:11,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:11,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:11,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:11,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:11,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:11,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,037 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:11,037 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:11,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:11,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:11,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360971134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,137 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:11,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:11,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:11,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:11,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:11,141 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:11,141 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,141 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:11,141 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:11,142 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,142 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:11,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:11,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:11,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:11,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:11,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:11,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:11,145 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:11,145 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,145 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:11,145 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:11,145 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:11,145 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,145 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:11,145 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:11,145 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,145 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:11,145 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:11,145 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:11,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:11,147 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,147 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:11,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:11,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e27b162[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:11,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:11,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,153 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401bbfc826baf4f49b18fcd0eb630a99669 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401bbfc826baf4f49b18fcd0eb630a99669 2023-04-01 14:55:11,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,156 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:11,156 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:11,156 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:11,158 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:11,158 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:11,158 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:11,158 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,159 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:11,159 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:11,159 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:11,159 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:11,159 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:11,159 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:11,159 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:11,161 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,161 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:11,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:11,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,163 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,166 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:11,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:11,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:11,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:11,167 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:11,167 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:11,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:11,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:11,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:11,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:11,267 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:11,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:11,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:11,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:11,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:11,272 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:11,272 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,272 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:11,272 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:11,273 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,273 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,275 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:11,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:11,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:11,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:11,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:11,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:11,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:11,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:11,276 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:11,276 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:11,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:11,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:11,277 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,277 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:11,277 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:11,277 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:11,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:11,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:11,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:11,278 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,278 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,279 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,279 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:11,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@daadd99[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:11,280 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:11,280 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,281 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,281 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:11,281 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:11,281 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,281 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:11,281 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:11,282 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:11,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,283 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:11,283 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:11,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:11,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:11,285 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:11,285 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:11,285 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:11,285 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:11,285 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:11,287 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,287 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,287 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:11,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:11,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:11,288 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:11,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:11,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:11,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:11,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:11,291 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:11,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:11,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:11,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:11,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:11,292 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:11,292 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:11,392 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:11,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:11,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:11,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:11,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:11,396 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:11,396 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,396 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:11,396 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:11,396 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,396 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,397 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:11,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:11,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:11,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:11,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:11,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:11,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:11,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:11,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:11,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:11,399 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,400 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:11,400 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:11,400 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:11,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:11,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:11,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:11,401 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,401 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,402 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,402 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:11,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:11,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dac194a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:11,403 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:11,403 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,405 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:11,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:11,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:11,405 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,405 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:11,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,407 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,407 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:11,407 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:11,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:11,409 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:11,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:11,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:11,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:11,409 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:11,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:11,411 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,413 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,420 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:11,420 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,420 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:11,420 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:11,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:11,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:11,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:11,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:11,420 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,421 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,421 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,421 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,421 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,421 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,421 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:11,421 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:11,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:11,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:11,437 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 14:55:11,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:11,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360971439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,521 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:11,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:11,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:11,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:11,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:11,524 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:11,524 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,524 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:11,524 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:11,525 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,525 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,526 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,526 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:11,526 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,527 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:11,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:11,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1680360971527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:11,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:11,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:11,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:11,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:11,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:11,532 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:11,532 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360971528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,532 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:11,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:11,532 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:11,532 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:11,532 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360971528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:11,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:11,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:11,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:11,534 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,534 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,535 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,535 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360971533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:11,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7315ebf[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:11,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:11,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,539 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:11,539 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,539 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:11,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:11,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:11,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:11,551 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:11,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:11,554 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:11,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:11,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:11,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:11,554 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:11,555 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,556 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,556 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:11,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:11,556 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:11,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:11,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:11,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,561 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/da9c137e305e45a8be27cfc7b52ef247 2023-04-01 14:55:11,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:11,563 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:11,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:11,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:11,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:11,564 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:11,564 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:11,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,564 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,564 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,564 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:11,564 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:11,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:11,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:11,584 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/a38817ccce0f40c7bd7f033cee74b3f1 2023-04-01 14:55:11,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/ae66db29b58145f2bef72378dec5e975 2023-04-01 14:55:11,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/da9c137e305e45a8be27cfc7b52ef247 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/da9c137e305e45a8be27cfc7b52ef247 2023-04-01 14:55:11,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/da9c137e305e45a8be27cfc7b52ef247, entries=200, sequenceid=51, filesize=31.9 K 2023-04-01 14:55:11,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/a38817ccce0f40c7bd7f033cee74b3f1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a38817ccce0f40c7bd7f033cee74b3f1 2023-04-01 14:55:11,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a38817ccce0f40c7bd7f033cee74b3f1, entries=150, sequenceid=51, filesize=11.6 K 2023-04-01 14:55:11,618 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/ae66db29b58145f2bef72378dec5e975 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ae66db29b58145f2bef72378dec5e975 2023-04-01 14:55:11,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ae66db29b58145f2bef72378dec5e975, entries=150, sequenceid=51, filesize=11.6 K 2023-04-01 14:55:11,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b2dc695f920ae42c5964c2b96bd0ec47 in 892ms, sequenceid=51, compaction requested=true 2023-04-01 14:55:11,622 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:11,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:11,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:11,622 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:11,622 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:11,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:11,623 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:11,623 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/B is initiating minor compaction (all files) 2023-04-01 14:55:11,623 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98062 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:11,624 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/B in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,624 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/A is initiating minor compaction (all files) 2023-04-01 14:55:11,624 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/efedb47336ff4e20ad763f998fb9b24b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a6b6c88c5307458bb8894230c256ffea, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a38817ccce0f40c7bd7f033cee74b3f1] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=34.8 K 2023-04-01 14:55:11,624 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/A in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,624 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/dbf51ab2b2984b28b3227da93b3da425, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/0abc269841ad4779a645e6a4b47a8f9c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/da9c137e305e45a8be27cfc7b52ef247] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=95.8 K 2023-04-01 14:55:11,624 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting efedb47336ff4e20ad763f998fb9b24b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680360910063 2023-04-01 14:55:11,624 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting dbf51ab2b2984b28b3227da93b3da425, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680360910063 2023-04-01 14:55:11,624 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a6b6c88c5307458bb8894230c256ffea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680360910087 2023-04-01 14:55:11,624 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 0abc269841ad4779a645e6a4b47a8f9c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680360910087 2023-04-01 14:55:11,625 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a38817ccce0f40c7bd7f033cee74b3f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680360910404 2023-04-01 14:55:11,625 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting da9c137e305e45a8be27cfc7b52ef247, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680360910398 2023-04-01 14:55:11,631 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#B#compaction#303 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:11,638 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#A#compaction#304 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:11,664 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:11,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:11,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:11,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:11,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:11,667 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:11,667 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,667 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:11,667 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:11,667 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,667 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,669 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:11,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:11,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:11,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:11,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:11,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:11,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:11,670 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:11,670 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:11,670 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:11,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:11,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:11,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:11,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:11,671 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:55:11,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:11,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:11,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:11,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:11,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:11,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:11,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:11,950 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:12,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:12,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360972007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,043 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/0f88d4bfe3004166bb80a20de2f78764 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/0f88d4bfe3004166bb80a20de2f78764 2023-04-01 14:55:12,048 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/B of b2dc695f920ae42c5964c2b96bd0ec47 into 0f88d4bfe3004166bb80a20de2f78764(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:12,048 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:12,048 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/B, priority=13, startTime=1680360911622; duration=0sec 2023-04-01 14:55:12,048 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:12,048 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:12,049 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:12,050 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/C is initiating minor compaction (all files) 2023-04-01 14:55:12,050 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/C in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:12,050 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bf202651dc994e73b2ab8f84b1976654, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9175178b572b4718baa688e694332d48, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ae66db29b58145f2bef72378dec5e975] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=34.8 K 2023-04-01 14:55:12,050 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting bf202651dc994e73b2ab8f84b1976654, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680360910063 2023-04-01 14:55:12,050 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9175178b572b4718baa688e694332d48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680360910087 2023-04-01 14:55:12,051 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ae66db29b58145f2bef72378dec5e975, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680360910404 2023-04-01 14:55:12,065 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#C#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:12,084 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401ff0872a37bf34fbd80e1fb48706d2a6a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ff0872a37bf34fbd80e1fb48706d2a6a 2023-04-01 14:55:12,091 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/7ae20c29ee57446983e6276b4d27d458 2023-04-01 14:55:12,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:12,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360972109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,117 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/2b453c9a589e4b79a4b63be64636120b 2023-04-01 14:55:12,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:12,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360972316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,480 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/9e8e202bb31f4cb79ef21c240391fa44 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9e8e202bb31f4cb79ef21c240391fa44 2023-04-01 14:55:12,484 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/C of b2dc695f920ae42c5964c2b96bd0ec47 into 9e8e202bb31f4cb79ef21c240391fa44(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:12,484 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:12,484 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/C, priority=13, startTime=1680360911622; duration=0sec 2023-04-01 14:55:12,484 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:12,529 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/bdf34fc4f3ce41b1a182ba7fb71bc9af 2023-04-01 14:55:12,533 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/7ae20c29ee57446983e6276b4d27d458 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7ae20c29ee57446983e6276b4d27d458 2023-04-01 14:55:12,537 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7ae20c29ee57446983e6276b4d27d458, entries=150, sequenceid=69, filesize=25.1 K 2023-04-01 14:55:12,538 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/2b453c9a589e4b79a4b63be64636120b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2b453c9a589e4b79a4b63be64636120b 2023-04-01 14:55:12,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:12,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360972536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:12,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1680360972538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:12,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360972540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:12,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360972542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,542 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2b453c9a589e4b79a4b63be64636120b, entries=150, sequenceid=69, filesize=11.6 K 2023-04-01 14:55:12,548 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/bdf34fc4f3ce41b1a182ba7fb71bc9af as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bdf34fc4f3ce41b1a182ba7fb71bc9af 2023-04-01 14:55:12,551 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bdf34fc4f3ce41b1a182ba7fb71bc9af, entries=150, sequenceid=69, filesize=11.6 K 2023-04-01 14:55:12,552 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b2dc695f920ae42c5964c2b96bd0ec47 in 881ms, sequenceid=69, compaction requested=false 2023-04-01 14:55:12,553 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:12,553 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:12,555 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:12,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@243e439a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:12,558 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:12,558 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,559 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,559 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:12,559 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:12,559 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:12,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:12,560 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,560 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:12,560 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,560 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:12,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:12,563 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:12,563 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:12,563 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:12,563 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:12,563 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:12,565 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,565 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,566 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:12,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:12,566 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,568 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:12,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,574 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:12,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:12,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:12,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:12,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:12,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:12,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,574 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:12,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,574 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:12,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:12,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 14:55:12,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:12,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:12,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:12,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:12,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:12,624 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:12,675 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:12,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:12,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:12,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:12,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:12,683 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:12,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:12,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:12,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:12,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:12,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360972681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,685 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,685 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,685 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:12,686 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:12,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:12,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:12,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:12,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:12,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:12,688 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:12,688 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:12,688 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:12,688 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:12,689 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:12,690 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,690 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:12,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fa31491[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:12,692 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:12,692 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,693 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,693 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:12,693 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:12,693 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:12,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,694 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:12,695 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,696 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,696 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:12,696 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:12,698 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:12,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:12,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:12,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:12,698 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:12,699 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,699 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,699 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,701 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,702 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,706 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,706 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:12,706 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:12,706 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:12,706 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:12,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:12,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:12,706 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:12,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:12,706 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,707 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:12,707 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,707 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:12,707 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,707 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:12,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360972785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,807 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:12,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:12,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:12,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:12,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:12,811 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:12,811 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:12,811 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:12,811 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:12,811 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,811 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,816 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,816 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:12,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:12,816 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,816 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:12,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:12,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:12,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:12,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:12,818 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:12,818 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:12,818 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:12,818 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:12,818 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:12,818 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:12,818 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:12,818 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:12,818 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,818 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:12,818 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:12,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,819 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,820 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,820 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42f82466[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:12,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:12,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,822 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,822 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:12,823 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:12,823 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:12,823 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,823 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:12,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:12,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:12,826 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:12,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:12,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:12,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:12,826 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:12,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:12,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:12,829 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,831 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,833 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,833 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:12,833 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,833 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:12,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:12,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:12,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:12,833 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:12,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:12,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:12,833 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,833 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,833 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,833 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,833 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:12,833 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:12,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:12,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:12,888 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/d097dcacf5f442838ccf34feb1abd584 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d097dcacf5f442838ccf34feb1abd584 2023-04-01 14:55:12,891 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/A of b2dc695f920ae42c5964c2b96bd0ec47 into d097dcacf5f442838ccf34feb1abd584(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-01 14:55:12,892 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:12,892 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/A, priority=13, startTime=1680360911622; duration=1sec 2023-04-01 14:55:12,892 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:12,933 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:12,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:12,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:12,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:12,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:12,936 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:12,936 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:12,936 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:12,936 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:12,937 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,937 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,938 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,938 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:12,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:12,938 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,938 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:12,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:12,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:12,939 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:12,939 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:12,939 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:12,939 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:12,939 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:12,939 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:12,939 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:12,940 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:12,940 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:12,940 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:12,940 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:12,940 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,940 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:12,940 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:12,941 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,941 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,941 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,941 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:12,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f315e20[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:12,943 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:12,943 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,945 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:12,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:12,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:12,945 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,945 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:12,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,946 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:12,946 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,946 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:12,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:12,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:12,949 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:12,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:12,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:12,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:12,949 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:12,950 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,950 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:12,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,950 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:12,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:12,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:12,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:12,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,952 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,955 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:12,955 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:12,955 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:12,955 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:12,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:12,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:12,955 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,955 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:12,955 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:12,955 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,955 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:12,955 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:12,955 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:12,956 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:12,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:12,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:12,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360972992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,036 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304012eb85744af1b491d93488186acf31830 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012eb85744af1b491d93488186acf31830 2023-04-01 14:55:13,041 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=86, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/ef12c646d19c42e4bff25bc5666480cc 2023-04-01 14:55:13,056 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:13,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:13,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:13,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:13,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:13,059 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:13,059 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:13,059 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:13,059 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:13,059 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,059 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,061 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:13,061 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:13,061 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,061 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:13,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:13,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:13,063 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,063 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:13,063 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:13,063 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:13,063 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:13,065 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,065 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:13,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c5f6cf[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:13,067 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:13,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,067 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:13,069 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,069 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:13,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,070 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:13,070 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:13,070 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:13,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:13,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:13,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:13,073 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:13,074 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:13,074 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:13,074 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:13,074 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:13,077 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,077 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,077 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:13,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,082 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,082 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,083 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:13,083 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:13,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:13,083 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,083 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,083 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,083 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,083 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:13,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:13,083 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:13,083 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:13,083 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:13,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:13,183 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:13,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:13,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:13,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:13,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:13,187 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:13,187 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:13,187 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:13,187 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:13,187 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,187 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,191 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:13,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:13,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:13,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:13,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:13,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:13,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:13,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:13,193 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,193 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:13,193 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:13,193 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:13,195 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e256efa[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:13,197 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:13,197 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,198 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:13,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:13,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:13,199 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,199 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:13,200 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:13,200 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:13,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:13,202 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:13,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:13,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:13,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:13,202 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:13,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,204 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,207 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,210 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:13,210 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,210 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:13,210 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:13,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:13,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:13,210 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,210 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,210 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:13,211 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:13,211 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:13,210 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,211 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,211 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:13,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:13,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:13,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360973295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,311 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:13,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:13,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:13,314 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:13,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:13,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:13,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:13,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:13,315 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:13,316 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,316 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:13,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:13,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:13,317 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,317 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:13,317 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:13,317 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:13,319 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,319 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,319 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,319 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:13,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dfdbe4c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:13,320 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:13,320 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,321 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,322 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:13,322 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:13,322 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:13,322 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,322 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:13,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:13,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:13,326 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:13,326 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:13,326 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:13,326 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:13,326 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:13,328 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,328 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,328 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,329 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,332 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,334 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:13,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:13,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:13,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:13,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:13,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:13,334 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:13,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,334 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:13,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:13,440 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:13,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:13,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:13,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:13,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:13,443 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:13,444 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:13,444 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:13,444 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:13,444 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,444 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,445 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,445 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:13,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:13,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:13,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:13,447 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:13,447 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:13,447 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:13,447 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:13,448 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,448 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:13,448 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:13,448 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:13,450 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:13,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38783f8d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:13,452 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:13,452 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,453 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:13,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:13,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:13,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:13,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,455 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,455 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:13,455 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/9d5173f476ee4c8898810f3baaff6ea0 2023-04-01 14:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:13,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:13,458 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:13,458 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:13,458 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:13,458 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:13,458 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:13,460 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,460 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:13,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:13,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,460 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:13,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:13,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:13,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:13,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,461 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:13,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/0d380cf874884bc99cac2022a97be2d6 2023-04-01 14:55:13,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:13,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:13,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:13,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:13,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:13,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,466 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:13,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:13,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,467 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:13,467 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:13,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:13,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:13,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/ef12c646d19c42e4bff25bc5666480cc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/ef12c646d19c42e4bff25bc5666480cc 2023-04-01 14:55:13,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/ef12c646d19c42e4bff25bc5666480cc, entries=200, sequenceid=86, filesize=31.9 K 2023-04-01 14:55:13,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/9d5173f476ee4c8898810f3baaff6ea0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/9d5173f476ee4c8898810f3baaff6ea0 2023-04-01 14:55:13,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/9d5173f476ee4c8898810f3baaff6ea0, entries=150, sequenceid=86, filesize=11.6 K 2023-04-01 14:55:13,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/0d380cf874884bc99cac2022a97be2d6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/0d380cf874884bc99cac2022a97be2d6 2023-04-01 14:55:13,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/0d380cf874884bc99cac2022a97be2d6, entries=150, sequenceid=86, filesize=11.6 K 2023-04-01 14:55:13,484 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for b2dc695f920ae42c5964c2b96bd0ec47 in 861ms, sequenceid=86, compaction requested=true 2023-04-01 14:55:13,484 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:13,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:13,485 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:13,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:13,485 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:13,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:13,486 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84281 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:13,486 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/A is initiating minor compaction (all files) 2023-04-01 14:55:13,486 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/A in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,486 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:13,486 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d097dcacf5f442838ccf34feb1abd584, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7ae20c29ee57446983e6276b4d27d458, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/ef12c646d19c42e4bff25bc5666480cc] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=82.3 K 2023-04-01 14:55:13,486 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/B is initiating minor compaction (all files) 2023-04-01 14:55:13,486 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/B in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,486 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/0f88d4bfe3004166bb80a20de2f78764, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2b453c9a589e4b79a4b63be64636120b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/9d5173f476ee4c8898810f3baaff6ea0] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=34.9 K 2023-04-01 14:55:13,487 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d097dcacf5f442838ccf34feb1abd584, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680360910404 2023-04-01 14:55:13,487 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 0f88d4bfe3004166bb80a20de2f78764, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680360910404 2023-04-01 14:55:13,487 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 7ae20c29ee57446983e6276b4d27d458, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680360910800 2023-04-01 14:55:13,487 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2b453c9a589e4b79a4b63be64636120b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680360910800 2023-04-01 14:55:13,487 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting ef12c646d19c42e4bff25bc5666480cc, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680360911971 2023-04-01 14:55:13,488 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9d5173f476ee4c8898810f3baaff6ea0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680360912001 2023-04-01 14:55:13,494 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#B#compaction#312 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:13,496 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#A#compaction#313 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:13,512 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/8da1459824814933956bb16ec6657af4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8da1459824814933956bb16ec6657af4 2023-04-01 14:55:13,516 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/B of b2dc695f920ae42c5964c2b96bd0ec47 into 8da1459824814933956bb16ec6657af4(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:13,516 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:13,516 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/B, priority=13, startTime=1680360913485; duration=0sec 2023-04-01 14:55:13,516 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:13,516 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:13,517 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:13,517 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/C is initiating minor compaction (all files) 2023-04-01 14:55:13,517 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/C in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,517 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9e8e202bb31f4cb79ef21c240391fa44, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bdf34fc4f3ce41b1a182ba7fb71bc9af, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/0d380cf874884bc99cac2022a97be2d6] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=34.9 K 2023-04-01 14:55:13,518 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9e8e202bb31f4cb79ef21c240391fa44, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680360910404 2023-04-01 14:55:13,518 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting bdf34fc4f3ce41b1a182ba7fb71bc9af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680360910800 2023-04-01 14:55:13,518 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 0d380cf874884bc99cac2022a97be2d6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680360912001 2023-04-01 14:55:13,523 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#C#compaction#314 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:13,567 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:13,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:13,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:13,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:13,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:13,570 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:13,570 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:13,570 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:13,570 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:13,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,572 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:13,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:13,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:13,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:13,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:13,574 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:13,574 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:13,574 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:13,574 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:13,574 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 14:55:13,574 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:13,574 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:13,574 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:13,574 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:13,574 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:13,574 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:13,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:13,807 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:13,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:13,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360973850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,932 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/ed660f2936be49eb877d3422eb72aeb8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ed660f2936be49eb877d3422eb72aeb8 2023-04-01 14:55:13,936 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/C of b2dc695f920ae42c5964c2b96bd0ec47 into ed660f2936be49eb877d3422eb72aeb8(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:13,936 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:13,936 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/C, priority=13, startTime=1680360913485; duration=0sec 2023-04-01 14:55:13,936 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:13,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:13,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360973955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:13,986 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040136f361823d3c4d33ad52b7d311ea53a2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040136f361823d3c4d33ad52b7d311ea53a2 2023-04-01 14:55:13,990 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/7239114145f44127b1d5c914dce51bdb 2023-04-01 14:55:14,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360974161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,401 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/2f054f93f02b4a4c8f0e54571c363ee6 2023-04-01 14:55:14,412 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/270f0d9c80ef40b3b630f3556f0887ab 2023-04-01 14:55:14,415 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/7239114145f44127b1d5c914dce51bdb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7239114145f44127b1d5c914dce51bdb 2023-04-01 14:55:14,418 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7239114145f44127b1d5c914dce51bdb, entries=150, sequenceid=105, filesize=25.1 K 2023-04-01 14:55:14,419 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/2f054f93f02b4a4c8f0e54571c363ee6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2f054f93f02b4a4c8f0e54571c363ee6 2023-04-01 14:55:14,421 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2f054f93f02b4a4c8f0e54571c363ee6, entries=150, sequenceid=105, filesize=11.6 K 2023-04-01 14:55:14,422 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/270f0d9c80ef40b3b630f3556f0887ab as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/270f0d9c80ef40b3b630f3556f0887ab 2023-04-01 14:55:14,426 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/270f0d9c80ef40b3b630f3556f0887ab, entries=150, sequenceid=105, filesize=11.6 K 2023-04-01 14:55:14,426 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for b2dc695f920ae42c5964c2b96bd0ec47 in 852ms, sequenceid=105, compaction requested=false 2023-04-01 14:55:14,427 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:14,427 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:14,427 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:14,427 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:14,427 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,427 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:14,427 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:14,428 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:14,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:14,429 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,429 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:14,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:14,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e3dd5c3[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:14,431 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:14,431 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,436 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:14,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:14,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:14,437 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,437 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:14,438 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,438 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:14,438 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:14,438 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:14,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:14,442 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:14,442 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:14,442 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:14,442 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:14,442 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:14,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:14,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,444 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:14,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,445 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:14,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,449 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:14,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:14,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:14,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:14,451 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:14,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:14,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:14,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:14,451 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:14,451 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:14,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:14,472 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 14:55:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:14,503 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304013fdd44d537634d7aa6e1ab809659fda4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013fdd44d537634d7aa6e1ab809659fda4 2023-04-01 14:55:14,528 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=122, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/1de550009af14fad838a35d224c833e8 2023-04-01 14:55:14,551 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:14,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:14,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:14,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:14,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:14,564 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:14,564 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:14,564 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:14,564 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:14,564 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,564 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1680360974559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360974562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360974562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,566 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,566 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:14,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:14,567 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,567 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:14,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360974565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360974566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:14,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:14,584 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:14,608 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:14,608 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:14,589 DEBUG [Thread-2172] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:55:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:14,617 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:14,617 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:14,617 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:14,617 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:14,618 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:14,618 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,618 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:14,618 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:14,619 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,619 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:14,620 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,620 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:14,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:14,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bba2840[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:14,623 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:14,623 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,626 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:14,626 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:14,626 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:14,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:14,627 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:14,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:14,627 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:14,627 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:14,628 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:14,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:14,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:14,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:14,628 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:14,633 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,633 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:14,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,633 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:14,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:14,635 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,636 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:14,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:14,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:14,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:14,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:14,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:14,642 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:14,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:14,642 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:14,642 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:14,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360974667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360974667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360974674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360974677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,727 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/f709dc92feb9452eb4e7384a4e24c1ac as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/f709dc92feb9452eb4e7384a4e24c1ac 2023-04-01 14:55:14,735 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/A of b2dc695f920ae42c5964c2b96bd0ec47 into f709dc92feb9452eb4e7384a4e24c1ac(size=25.3 K), total size for store is 50.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-01 14:55:14,735 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:14,735 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/A, priority=13, startTime=1680360913485; duration=1sec 2023-04-01 14:55:14,735 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:14,742 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:14,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:14,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:14,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:14,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:14,755 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:14,755 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:14,755 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:14,755 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:14,756 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,757 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,758 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,758 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,758 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:14,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:14,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:14,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:14,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:14,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:14,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:14,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:14,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:14,771 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:14,771 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:14,771 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:14,771 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:14,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:14,773 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,773 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:14,774 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,774 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:14,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:14,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ba5f2a5[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:14,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:14,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,781 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,781 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:14,781 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:14,781 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:14,782 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,782 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:14,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,785 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:14,785 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:14,785 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:14,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:14,789 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:14,789 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:14,789 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:14,789 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:14,789 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:14,790 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:14,791 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,792 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,792 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:14,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:14,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,794 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:14,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,797 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:14,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:14,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:14,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:14,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:14,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:14,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:14,801 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:14,801 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:14,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360974874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360974874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360974879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:14,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360974885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,901 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:14,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:14,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:14,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:14,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:14,912 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:14,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:14,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:14,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:14,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,914 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:14,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:14,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:14,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:14,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:14,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:14,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:14,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:14,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:14,932 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:14,932 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:14,932 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:14,932 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:14,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:14,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:14,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:14,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:14,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:14,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:14,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3258280b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:14,945 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:14,945 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,946 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:14,946 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:14,946 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:14,947 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,947 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:14,951 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:14,951 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,951 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:14,951 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:14,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:14,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:14,955 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:14,955 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:14,955 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:14,955 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:14,955 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:14,957 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,957 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:14,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,957 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,958 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:14,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,961 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:14,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:14,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:14,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:14,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:14,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:14,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:14,965 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:14,966 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:14,966 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:14,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,966 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,966 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:14,966 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:14,966 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,966 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:14,966 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:14,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:14,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:14,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/3cabb2de6eaf4eda96d7022fe178f8e3 2023-04-01 14:55:14,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/ac93e911987444609a8e5acf03d11f25 2023-04-01 14:55:14,997 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/1de550009af14fad838a35d224c833e8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/1de550009af14fad838a35d224c833e8 2023-04-01 14:55:15,000 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/1de550009af14fad838a35d224c833e8, entries=200, sequenceid=122, filesize=31.9 K 2023-04-01 14:55:15,001 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/3cabb2de6eaf4eda96d7022fe178f8e3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/3cabb2de6eaf4eda96d7022fe178f8e3 2023-04-01 14:55:15,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/3cabb2de6eaf4eda96d7022fe178f8e3, entries=150, sequenceid=122, filesize=11.6 K 2023-04-01 14:55:15,007 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/ac93e911987444609a8e5acf03d11f25 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ac93e911987444609a8e5acf03d11f25 2023-04-01 14:55:15,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ac93e911987444609a8e5acf03d11f25, entries=150, sequenceid=122, filesize=11.6 K 2023-04-01 14:55:15,011 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b2dc695f920ae42c5964c2b96bd0ec47 in 539ms, sequenceid=122, compaction requested=true 2023-04-01 14:55:15,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:15,011 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:15,011 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:15,011 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:15,011 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:15,011 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:15,012 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84384 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:15,012 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/A is initiating minor compaction (all files) 2023-04-01 14:55:15,012 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:15,012 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/A in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,012 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/B is initiating minor compaction (all files) 2023-04-01 14:55:15,012 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/f709dc92feb9452eb4e7384a4e24c1ac, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7239114145f44127b1d5c914dce51bdb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/1de550009af14fad838a35d224c833e8] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=82.4 K 2023-04-01 14:55:15,012 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/B in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,012 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8da1459824814933956bb16ec6657af4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2f054f93f02b4a4c8f0e54571c363ee6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/3cabb2de6eaf4eda96d7022fe178f8e3] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=35.0 K 2023-04-01 14:55:15,013 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting f709dc92feb9452eb4e7384a4e24c1ac, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680360912001 2023-04-01 14:55:15,013 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 8da1459824814933956bb16ec6657af4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680360912001 2023-04-01 14:55:15,013 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 7239114145f44127b1d5c914dce51bdb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680360912653 2023-04-01 14:55:15,013 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2f054f93f02b4a4c8f0e54571c363ee6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680360912653 2023-04-01 14:55:15,014 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 1de550009af14fad838a35d224c833e8, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680360913820 2023-04-01 14:55:15,014 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3cabb2de6eaf4eda96d7022fe178f8e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680360913820 2023-04-01 14:55:15,021 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#B#compaction#322 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:15,031 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#A#compaction#321 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:15,043 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/8e641f4519144071828d4ad7981bd924 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8e641f4519144071828d4ad7981bd924 2023-04-01 14:55:15,047 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/B of b2dc695f920ae42c5964c2b96bd0ec47 into 8e641f4519144071828d4ad7981bd924(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:15,048 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:15,048 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/B, priority=13, startTime=1680360915011; duration=0sec 2023-04-01 14:55:15,048 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:15,048 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:15,049 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:15,049 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/C is initiating minor compaction (all files) 2023-04-01 14:55:15,049 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/C in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,049 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ed660f2936be49eb877d3422eb72aeb8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/270f0d9c80ef40b3b630f3556f0887ab, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ac93e911987444609a8e5acf03d11f25] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=35.0 K 2023-04-01 14:55:15,049 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ed660f2936be49eb877d3422eb72aeb8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1680360912001 2023-04-01 14:55:15,050 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 270f0d9c80ef40b3b630f3556f0887ab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680360912653 2023-04-01 14:55:15,050 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ac93e911987444609a8e5acf03d11f25, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680360913820 2023-04-01 14:55:15,055 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#C#compaction#323 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:15,066 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:15,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:15,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:15,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:15,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:15,076 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:15,076 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:15,076 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:15,076 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:15,077 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,077 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,078 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:15,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:15,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:15,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:15,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:15,079 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:15,079 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:15,080 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:15,080 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:15,080 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:15,080 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,080 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:15,080 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 14:55:15,080 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:15,081 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,081 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:15,081 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,081 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:15,081 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,115 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040188c27cd356414b67b1b57ee9545b5421 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040188c27cd356414b67b1b57ee9545b5421 2023-04-01 14:55:15,142 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=140, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/cc7425792f794ff09cf0d0a976a1f656 2023-04-01 14:55:15,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:15,186 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:15,197 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/8267a639285b4177a4eafe40113e8d58 2023-04-01 14:55:15,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360975208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360975221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360975225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360975228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,236 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/51263beccfc3489f969108b892b055c7 2023-04-01 14:55:15,249 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/cc7425792f794ff09cf0d0a976a1f656 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/cc7425792f794ff09cf0d0a976a1f656 2023-04-01 14:55:15,256 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/cc7425792f794ff09cf0d0a976a1f656, entries=150, sequenceid=140, filesize=25.3 K 2023-04-01 14:55:15,257 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/8267a639285b4177a4eafe40113e8d58 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8267a639285b4177a4eafe40113e8d58 2023-04-01 14:55:15,267 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8267a639285b4177a4eafe40113e8d58, entries=150, sequenceid=140, filesize=11.8 K 2023-04-01 14:55:15,268 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/51263beccfc3489f969108b892b055c7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/51263beccfc3489f969108b892b055c7 2023-04-01 14:55:15,281 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/51263beccfc3489f969108b892b055c7, entries=150, sequenceid=140, filesize=11.8 K 2023-04-01 14:55:15,282 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b2dc695f920ae42c5964c2b96bd0ec47 in 202ms, sequenceid=140, compaction requested=false 2023-04-01 14:55:15,282 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:15,282 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:15,283 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,283 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:15,283 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:15,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,285 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:15,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52ab704e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:15,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:15,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,289 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:15,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,289 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:15,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,291 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:15,291 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:15,291 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:15,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:15,294 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:15,294 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:15,294 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:15,294 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:15,294 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:15,296 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,296 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:15,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,296 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:15,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:15,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,297 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:15,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:15,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:15,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:15,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:15,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:15,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:15,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,303 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:15,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:15,303 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:15,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:15,350 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 14:55:15,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:15,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:15,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:15,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,388 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401bb7a20ae0f54410cab3304dbbbdd0dc6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401bb7a20ae0f54410cab3304dbbbdd0dc6 2023-04-01 14:55:15,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360975386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43524 deadline: 1680360975386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360975387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360975390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,403 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:15,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:15,404 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/d4ca18befccc44f9b23066ec0c46753a 2023-04-01 14:55:15,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:15,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:15,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:15,408 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:15,408 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:15,408 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:15,408 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:15,408 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,412 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:15,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:15,412 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,413 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,413 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:15,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:15,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:15,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:15,428 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,428 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:15,428 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:15,428 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:15,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:15,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:15,433 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:15,434 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,434 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:15,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:15,435 WARN [DataStreamer for file /user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/20b52a782945484f8a7a89d35059d64a] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-01 14:55:15,436 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:15,436 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/20b52a782945484f8a7a89d35059d64a 2023-04-01 14:55:15,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d9aa73[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:15,437 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,437 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:15,437 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:15,437 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:15,438 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,438 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:15,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:15,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:15,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:15,440 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:15,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:15,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:15,445 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:15,445 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:15,445 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:15,445 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:15,445 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:15,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:15,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:15,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:15,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:15,448 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,451 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,456 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:15,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:15,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:15,457 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:15,457 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:15,457 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:15,457 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:15,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:15,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:15,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/c40306e4754a4644a16b23db54ea8a52 2023-04-01 14:55:15,475 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/3b145bf88ec64bef9276cc9bc3ad10a8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/3b145bf88ec64bef9276cc9bc3ad10a8 2023-04-01 14:55:15,479 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/5897f6631ee5466eb40f81f3af7ccdb1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/5897f6631ee5466eb40f81f3af7ccdb1 2023-04-01 14:55:15,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/d4ca18befccc44f9b23066ec0c46753a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d4ca18befccc44f9b23066ec0c46753a 2023-04-01 14:55:15,485 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/A of b2dc695f920ae42c5964c2b96bd0ec47 into 3b145bf88ec64bef9276cc9bc3ad10a8(size=25.4 K), total size for store is 50.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:15,485 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:15,485 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/A, priority=13, startTime=1680360915011; duration=0sec 2023-04-01 14:55:15,485 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:15,487 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/C of b2dc695f920ae42c5964c2b96bd0ec47 into 5897f6631ee5466eb40f81f3af7ccdb1(size=11.9 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:15,487 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:15,487 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/C, priority=13, startTime=1680360915011; duration=0sec 2023-04-01 14:55:15,487 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:15,487 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d4ca18befccc44f9b23066ec0c46753a, entries=150, sequenceid=157, filesize=25.3 K 2023-04-01 14:55:15,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/20b52a782945484f8a7a89d35059d64a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/20b52a782945484f8a7a89d35059d64a 2023-04-01 14:55:15,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/20b52a782945484f8a7a89d35059d64a, entries=100, sequenceid=157, filesize=9.4 K 2023-04-01 14:55:15,492 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/c40306e4754a4644a16b23db54ea8a52 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c40306e4754a4644a16b23db54ea8a52 2023-04-01 14:55:15,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c40306e4754a4644a16b23db54ea8a52, entries=100, sequenceid=157, filesize=9.4 K 2023-04-01 14:55:15,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for b2dc695f920ae42c5964c2b96bd0ec47 in 146ms, sequenceid=157, compaction requested=true 2023-04-01 14:55:15,496 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:15,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:15,496 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:15,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:15,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 14:55:15,496 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:15,498 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77842 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:15,498 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/A is initiating minor compaction (all files) 2023-04-01 14:55:15,498 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/A in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,498 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/3b145bf88ec64bef9276cc9bc3ad10a8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/cc7425792f794ff09cf0d0a976a1f656, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d4ca18befccc44f9b23066ec0c46753a] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=76.0 K 2023-04-01 14:55:15,498 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33872 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:15,498 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/B is initiating minor compaction (all files) 2023-04-01 14:55:15,498 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/B in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,498 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 3b145bf88ec64bef9276cc9bc3ad10a8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680360913820 2023-04-01 14:55:15,498 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8e641f4519144071828d4ad7981bd924, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8267a639285b4177a4eafe40113e8d58, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/20b52a782945484f8a7a89d35059d64a] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=33.1 K 2023-04-01 14:55:15,499 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting cc7425792f794ff09cf0d0a976a1f656, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680360914523 2023-04-01 14:55:15,499 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 8e641f4519144071828d4ad7981bd924, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680360913820 2023-04-01 14:55:15,501 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d4ca18befccc44f9b23066ec0c46753a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680360915205 2023-04-01 14:55:15,501 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 8267a639285b4177a4eafe40113e8d58, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680360914523 2023-04-01 14:55:15,501 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 20b52a782945484f8a7a89d35059d64a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680360915205 2023-04-01 14:55:15,507 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#B#compaction#331 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:15,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:15,514 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 14:55:15,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:15,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:15,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:15,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,539 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#A#compaction#330 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:15,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360975539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360975539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360975546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,557 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:15,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:15,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360975550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:15,561 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/166931bac05540a89868ec939a57aea0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/166931bac05540a89868ec939a57aea0 2023-04-01 14:55:15,571 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/B of b2dc695f920ae42c5964c2b96bd0ec47 into 166931bac05540a89868ec939a57aea0(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:15,571 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:15,571 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/B, priority=13, startTime=1680360915496; duration=0sec 2023-04-01 14:55:15,571 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:15,571 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:15,573 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33872 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:15,573 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/C is initiating minor compaction (all files) 2023-04-01 14:55:15,574 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/C in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,574 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/5897f6631ee5466eb40f81f3af7ccdb1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/51263beccfc3489f969108b892b055c7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c40306e4754a4644a16b23db54ea8a52] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=33.1 K 2023-04-01 14:55:15,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:15,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:15,574 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:15,574 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:15,574 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:15,574 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:15,575 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 5897f6631ee5466eb40f81f3af7ccdb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1680360913820 2023-04-01 14:55:15,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,576 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 51263beccfc3489f969108b892b055c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680360914523 2023-04-01 14:55:15,576 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,576 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:15,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:15,578 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c40306e4754a4644a16b23db54ea8a52, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680360915205 2023-04-01 14:55:15,578 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,578 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:15,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:15,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:15,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:15,597 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:15,597 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:15,597 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:15,602 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:15,603 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:15,603 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,603 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:15,603 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:15,603 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,604 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:15,604 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:15,604 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,613 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#C#compaction#333 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:15,619 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,619 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:15,619 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304014c1453852413498580001bf5f3f04722 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304014c1453852413498580001bf5f3f04722 2023-04-01 14:55:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,620 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:15,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:15,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c8d5b17[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:15,622 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:15,622 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,623 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,624 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:15,624 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:15,624 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:15,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:15,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,626 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:15,626 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:15,626 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:15,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:15,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:15,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:15,628 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:15,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:15,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:15,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:15,628 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:15,631 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:15,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:15,631 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,631 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:15,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:15,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:15,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:15,633 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:15,635 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:15,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,644 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,644 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:15,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:15,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:15,643 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:15,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:15,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:15,644 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,643 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:15,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:15,644 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:15,644 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:15,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:15,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360975651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360975652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360975654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,660 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=178, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/9312c3bfd43745de8db6530804a9c0d6 2023-04-01 14:55:15,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360975661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,688 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/3c6294c262534eb58262d8def66d4281 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/3c6294c262534eb58262d8def66d4281 2023-04-01 14:55:15,693 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/C of b2dc695f920ae42c5964c2b96bd0ec47 into 3c6294c262534eb58262d8def66d4281(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:15,693 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:15,693 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/C, priority=13, startTime=1680360915496; duration=0sec 2023-04-01 14:55:15,693 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:15,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/0eac0a24f1a946d799b8a94b10ec0701 2023-04-01 14:55:15,733 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/483fe001c759453b948fa0a8451cbba0 2023-04-01 14:55:15,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/9312c3bfd43745de8db6530804a9c0d6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9312c3bfd43745de8db6530804a9c0d6 2023-04-01 14:55:15,741 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9312c3bfd43745de8db6530804a9c0d6, entries=350, sequenceid=178, filesize=52.6 K 2023-04-01 14:55:15,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/0eac0a24f1a946d799b8a94b10ec0701 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/0eac0a24f1a946d799b8a94b10ec0701 2023-04-01 14:55:15,744 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:15,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:15,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:15,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/0eac0a24f1a946d799b8a94b10ec0701, entries=150, sequenceid=178, filesize=11.8 K 2023-04-01 14:55:15,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:15,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:15,748 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:15,748 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:15,748 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:15,748 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:15,749 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,749 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/483fe001c759453b948fa0a8451cbba0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/483fe001c759453b948fa0a8451cbba0 2023-04-01 14:55:15,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,751 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:15,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:15,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/483fe001c759453b948fa0a8451cbba0, entries=150, sequenceid=178, filesize=11.8 K 2023-04-01 14:55:15,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:15,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for b2dc695f920ae42c5964c2b96bd0ec47 in 240ms, sequenceid=178, compaction requested=false 2023-04-01 14:55:15,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:15,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:15,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:15,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:15,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:15,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:15,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:15,755 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:15,755 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:15,755 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:15,755 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:15,756 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 14:55:15,756 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:15,756 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,756 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:15,756 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,756 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:15,756 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:15,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:15,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:15,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360975882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360975884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360975885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360975888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360975989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360975992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360975992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:15,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:15,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360975992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:16,173 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304011344f6c6db324a5f8e7bd229516ec2fc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011344f6c6db324a5f8e7bd229516ec2fc 2023-04-01 14:55:16,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:16,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360976194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:16,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:16,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360976195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:16,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:16,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360976196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:16,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:16,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360976198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:16,469 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/02e43b49d272474b90b86a7a9ca3dd4c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/02e43b49d272474b90b86a7a9ca3dd4c 2023-04-01 14:55:16,474 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/A of b2dc695f920ae42c5964c2b96bd0ec47 into 02e43b49d272474b90b86a7a9ca3dd4c(size=25.7 K), total size for store is 78.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:16,474 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:16,474 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/A, priority=13, startTime=1680360915496; duration=0sec 2023-04-01 14:55:16,474 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:16,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:16,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360976501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:16,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:16,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360976502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:16,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:16,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360976505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:16,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:16,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360976507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:16,579 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/5e1ef2fc8cec431c87bf46e37ec9a988 2023-04-01 14:55:16,990 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/2dd8268f77274b09ba85f0b79cf62e94 2023-04-01 14:55:17,003 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/c7848e140bf24b46ab755fb8933d29ba 2023-04-01 14:55:17,008 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/5e1ef2fc8cec431c87bf46e37ec9a988 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/5e1ef2fc8cec431c87bf46e37ec9a988 2023-04-01 14:55:17,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:17,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360977007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:17,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360977009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:17,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360977009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,014 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/5e1ef2fc8cec431c87bf46e37ec9a988, entries=150, sequenceid=193, filesize=25.3 K 2023-04-01 14:55:17,015 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/2dd8268f77274b09ba85f0b79cf62e94 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2dd8268f77274b09ba85f0b79cf62e94 2023-04-01 14:55:17,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:17,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360977015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,019 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2dd8268f77274b09ba85f0b79cf62e94, entries=150, sequenceid=193, filesize=11.8 K 2023-04-01 14:55:17,020 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/c7848e140bf24b46ab755fb8933d29ba as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c7848e140bf24b46ab755fb8933d29ba 2023-04-01 14:55:17,025 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c7848e140bf24b46ab755fb8933d29ba, entries=150, sequenceid=193, filesize=11.8 K 2023-04-01 14:55:17,025 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for b2dc695f920ae42c5964c2b96bd0ec47 in 1269ms, sequenceid=193, compaction requested=true 2023-04-01 14:55:17,026 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:17,026 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:17,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:17,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:17,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:17,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:17,027 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,027 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:17,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:17,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f6624fd[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:17,031 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:17,031 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:17,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:17,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:17,032 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:17,032 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:17,032 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:17,032 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,032 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:17,039 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:17,039 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:17,039 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:17,041 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:17,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:17,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:17,047 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:17,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:17,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:17,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:17,047 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:17,049 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:17,049 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:17,049 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:17,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:17,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:17,050 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:17,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:17,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:17,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:17,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:17,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:17,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:17,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:17,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:17,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:17,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:17,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:17,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:17,055 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:17,055 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:17,155 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:17,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:17,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:17,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:17,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:17,158 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:17,158 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:17,158 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:17,158 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:17,159 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:17,159 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:17,160 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:17,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:17,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:17,160 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,160 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:17,160 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:17,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:17,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:17,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:17,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:17,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:17,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:17,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:17,161 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:17,161 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:17,161 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:17,163 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:17,163 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:17,163 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:17,163 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:17,164 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:55:17,164 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:17,164 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:17,164 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:17,164 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:17,164 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:17,165 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:17,203 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401f413cef780ba40ff8c7937b3586b61c0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f413cef780ba40ff8c7937b3586b61c0 2023-04-01 14:55:17,619 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=213, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/6ddf3c6321d24f6db4f1b7bfb70f5db9 2023-04-01 14:55:18,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:18,023 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:18,028 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/a1a9faa70b114fbb8de847113aa8faa2 2023-04-01 14:55:18,045 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/9b5f76bd4dc44149836f2fc753e42a1f 2023-04-01 14:55:18,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360978038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360978039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,049 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/6ddf3c6321d24f6db4f1b7bfb70f5db9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6ddf3c6321d24f6db4f1b7bfb70f5db9 2023-04-01 14:55:18,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360978046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,053 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6ddf3c6321d24f6db4f1b7bfb70f5db9, entries=150, sequenceid=213, filesize=25.3 K 2023-04-01 14:55:18,053 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/a1a9faa70b114fbb8de847113aa8faa2 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a1a9faa70b114fbb8de847113aa8faa2 2023-04-01 14:55:18,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360978047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,059 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a1a9faa70b114fbb8de847113aa8faa2, entries=150, sequenceid=213, filesize=11.8 K 2023-04-01 14:55:18,060 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/9b5f76bd4dc44149836f2fc753e42a1f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9b5f76bd4dc44149836f2fc753e42a1f 2023-04-01 14:55:18,067 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9b5f76bd4dc44149836f2fc753e42a1f, entries=150, sequenceid=213, filesize=11.8 K 2023-04-01 14:55:18,068 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for b2dc695f920ae42c5964c2b96bd0ec47 in 904ms, sequenceid=213, compaction requested=true 2023-04-01 14:55:18,068 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:18,068 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:18,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,072 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,072 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,072 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4646045f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:18,075 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:18,075 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,078 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:18,080 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:18,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,080 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,080 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:18,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:18,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:18,080 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:18,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:18,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:18,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:18,080 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:18,082 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,082 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,082 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,083 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,086 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,090 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,090 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,090 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,090 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,090 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,090 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:18,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:18,090 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:18,090 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,091 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:18,091 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:18,090 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,091 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:18,152 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:18,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:18,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:18,153 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:18,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:18,153 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:18,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:18,191 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:18,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:18,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360978180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360978181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360978193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:18,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360978193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:18,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:18,195 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:18,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:18,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:18,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,196 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,196 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,196 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,196 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:18,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:18,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:18,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:18,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:18,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:18,198 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:18,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,199 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:18,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53d9b4d6[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:18,201 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:18,201 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,202 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:18,202 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:18,202 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:18,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:18,204 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:18,204 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,204 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:18,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:18,206 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:18,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:18,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:18,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:18,206 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:18,208 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,208 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,208 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,209 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,211 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:18,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:18,216 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,216 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:18,216 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:18,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:18,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360978294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360978294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360978294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360978294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,316 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:18,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:18,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:18,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:18,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:18,319 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:18,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:18,319 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:18,320 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,320 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,325 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,325 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,325 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,325 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:18,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:18,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:18,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:18,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:18,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:18,327 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,327 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:18,327 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:18,327 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:18,327 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:18,328 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,328 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,329 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,329 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:18,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:18,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55a0859[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:18,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,332 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:18,332 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,332 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:18,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:18,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:18,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:18,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:18,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:18,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:18,336 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:18,336 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:18,336 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:18,336 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:18,336 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:18,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,340 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,341 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,343 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,343 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,343 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:18,343 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:18,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:18,343 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,343 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,344 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,344 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:18,344 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:18,344 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,344 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,344 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,444 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:18,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:18,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:18,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:18,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:18,447 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:18,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:18,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:18,448 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,448 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,449 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:18,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:18,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:18,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:18,450 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:18,450 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,450 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:18,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:18,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:18,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:18,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:18,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:18,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:18,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:18,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:18,452 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,452 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:18,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:18,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b6595c3[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:18,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:18,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,456 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,456 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:18,459 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,459 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:18,459 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:18,459 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:18,461 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:18,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:18,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:18,463 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:18,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:18,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:18,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:18,463 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:18,465 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,465 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,465 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,466 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,469 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:18,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:18,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,469 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:18,470 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:18,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360978501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360978501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360978502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360978502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,570 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:18,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:18,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:18,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:18,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:18,573 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401da40e695c24248df98cb25de9b3bf05a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401da40e695c24248df98cb25de9b3bf05a 2023-04-01 14:55:18,573 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:18,573 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,573 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:18,574 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:18,574 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,574 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,576 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,576 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,577 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,577 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:18,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:18,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:18,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:18,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:18,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:18,578 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:18,578 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,578 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:18,578 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:18,578 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:18,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:18,581 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:18,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d77120c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:18,583 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:18,583 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,584 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:18,584 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:18,587 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:18,587 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,587 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:18,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:18,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:18,587 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:18,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:18,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:18,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:18,587 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:18,590 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,590 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,590 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,594 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:18,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:18,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,594 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:18,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,594 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:18,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1680360978643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,651 DEBUG [Thread-2172] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8243 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:55:18,695 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:18,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:18,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:18,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:18,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:18,697 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:18,697 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:18,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:18,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,702 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,702 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,703 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,703 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:18,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:18,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:18,704 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,704 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:18,704 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:18,704 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:18,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:18,707 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,707 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,707 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,707 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:18,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13967870[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:18,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:18,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,711 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,711 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:18,711 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:18,711 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:18,711 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,711 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:18,712 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:18,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:18,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:18,717 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:18,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:18,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:18,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:18,718 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:18,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,720 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,721 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,724 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,724 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,724 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,724 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:18,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:18,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:18,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,724 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,724 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,724 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:18,724 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,724 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:18,725 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,725 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,725 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360978806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360978808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360978808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:18,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360978809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,825 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:18,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:18,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:18,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:18,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:18,827 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:18,828 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,828 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:18,828 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:18,828 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,828 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,829 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,829 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,829 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:18,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:18,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:18,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:18,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,830 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:18,831 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:18,831 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:18,831 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,831 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:18,831 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:18,831 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,831 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:18,831 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:18,831 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,831 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:18,831 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:18,832 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,833 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:18,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46dba29b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:18,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:18,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,835 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,835 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,835 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:18,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:18,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:18,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:18,837 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:18,837 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,837 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:18,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:18,840 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:18,840 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:18,840 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:18,840 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:18,840 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:18,842 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,842 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,842 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,842 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,843 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,845 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,845 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,845 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,845 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,845 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:18,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:18,845 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,846 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:18,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:18,845 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,846 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:18,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,846 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,946 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:18,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:18,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:18,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:18,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:18,949 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:18,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:18,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:18,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,951 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:18,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:18,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:18,953 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,953 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:18,953 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:18,953 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:18,955 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,956 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,956 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,956 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:18,957 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:18,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11c7ec0e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:18,957 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,958 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,958 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:18,958 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:18,958 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:18,959 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,959 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:18,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:18,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:18,962 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:18,962 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:18,962 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:18,962 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:18,962 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:18,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:18,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:18,964 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,966 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,973 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,973 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:18,973 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:18,973 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:18,973 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:18,974 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:18,974 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,974 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:18,974 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:18,973 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:18,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:18,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:18,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:18,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:18,974 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:18,974 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:18,979 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=227, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/44daaa7b19e4421fa8ca97910031e069 2023-04-01 14:55:19,074 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:19,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:19,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:19,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:19,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:19,077 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:19,077 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,077 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:19,077 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:19,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,079 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,079 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:19,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:19,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:19,081 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,081 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:19,081 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:19,081 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:19,082 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:19,085 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,086 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,086 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:19,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:19,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24156b48[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:19,087 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:19,088 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:19,090 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,090 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:19,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:19,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:19,091 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,091 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:19,091 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,091 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:19,096 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:19,096 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:19,096 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:19,096 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:19,096 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:19,097 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,097 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,098 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,098 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,100 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,102 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,102 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,102 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:19,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:19,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,103 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:19,103 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:19,102 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,102 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,102 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,203 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:19,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:19,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:19,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:19,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:19,206 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:19,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:19,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:19,207 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,207 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,209 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,209 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,209 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,209 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:19,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:19,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:19,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:19,210 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:19,210 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,210 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:19,210 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:19,210 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:19,210 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,210 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:19,211 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:19,211 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:19,212 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e0d242[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:19,215 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:19,215 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,216 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,216 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:19,216 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:19,216 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:19,216 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,216 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:19,217 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,217 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:19,217 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,217 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:19,219 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:19,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:19,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:19,219 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:19,219 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:19,222 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,222 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,222 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,223 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,224 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,226 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:19,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:19,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:19,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,226 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:19,226 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:19,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:19,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360979312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:19,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360979314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:19,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360979316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:19,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360979316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,327 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:19,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:19,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:19,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:19,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:19,330 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:19,330 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,330 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:19,330 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:19,330 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,330 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,331 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,332 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,332 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:19,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:19,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:19,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:19,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:19,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:19,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:19,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:19,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:19,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:19,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:19,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,333 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:19,335 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,335 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,335 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,335 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fb4673f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:19,337 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:19,337 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:19,339 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,339 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:19,341 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:19,341 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,341 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,341 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:19,343 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:19,343 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:19,343 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:19,343 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:19,343 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:19,344 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,344 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,345 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,346 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,348 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,348 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,348 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:19,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:19,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:19,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,349 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,349 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:19,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,349 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:19,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/5615cff9c4134ea8a29fe797d4622086 2023-04-01 14:55:19,449 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:19,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:19,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:19,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:19,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:19,452 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:19,453 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,453 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:19,453 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:19,453 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,453 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,455 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,455 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,456 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,456 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:19,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:19,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:19,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:19,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:19,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:19,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:19,457 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,457 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:19,457 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:19,457 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:19,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:19,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:19,458 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:19,461 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:19,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31bd4f5b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:19,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:19,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:19,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:19,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:19,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:19,478 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:19,478 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,478 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,478 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:19,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:19,480 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:19,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:19,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:19,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:19,480 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:19,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,482 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,483 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,485 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,486 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:19,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:19,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,486 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:19,486 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:19,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,586 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:19,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:19,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:19,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:19,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:19,590 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:19,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:19,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:19,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,592 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,592 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:19,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:19,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:19,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:19,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:19,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:19,593 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,593 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:19,593 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:19,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:19,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:19,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:19,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:19,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:19,597 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3efd2d0f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:19,599 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:19,599 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:19,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:19,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:19,600 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,600 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:19,601 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,601 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:19,602 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,602 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:19,604 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:19,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:19,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:19,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:19,604 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:19,605 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,605 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,605 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,606 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,607 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,614 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,614 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,614 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,614 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,614 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:19,614 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:19,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:19,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,615 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:19,615 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:19,715 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:19,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:19,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:19,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:19,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:19,718 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:19,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:19,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:19,719 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,719 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,720 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,720 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,721 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,721 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:19,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:19,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:19,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:19,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:19,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:19,722 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,722 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:19,722 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:19,722 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:19,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:19,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:19,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:19,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,724 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c8d9a50[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:19,726 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:19,726 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,727 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,727 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:19,727 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:19,727 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:19,727 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,727 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:19,729 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,729 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:19,729 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,729 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:19,731 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:19,731 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:19,731 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:19,731 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:19,731 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:19,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:19,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:19,733 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,734 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,736 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,736 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:19,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,736 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:19,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:19,736 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,737 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,737 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,736 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:19,737 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,737 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:19,737 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:19,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:19,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:19,737 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:19,737 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,798 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/9058932546d14a20978fc9308a0eb8f5 2023-04-01 14:55:19,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/44daaa7b19e4421fa8ca97910031e069 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/44daaa7b19e4421fa8ca97910031e069 2023-04-01 14:55:19,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/44daaa7b19e4421fa8ca97910031e069, entries=200, sequenceid=227, filesize=32.1 K 2023-04-01 14:55:19,805 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/5615cff9c4134ea8a29fe797d4622086 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/5615cff9c4134ea8a29fe797d4622086 2023-04-01 14:55:19,808 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/5615cff9c4134ea8a29fe797d4622086, entries=150, sequenceid=227, filesize=11.8 K 2023-04-01 14:55:19,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/9058932546d14a20978fc9308a0eb8f5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9058932546d14a20978fc9308a0eb8f5 2023-04-01 14:55:19,811 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9058932546d14a20978fc9308a0eb8f5, entries=150, sequenceid=227, filesize=11.8 K 2023-04-01 14:55:19,812 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for b2dc695f920ae42c5964c2b96bd0ec47 in 1660ms, sequenceid=227, compaction requested=true 2023-04-01 14:55:19,812 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:19,812 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:19,812 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:55:19,812 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:55:19,813 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:19,813 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:19,814 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60590 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:55:19,814 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/B is initiating minor compaction (all files) 2023-04-01 14:55:19,814 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 164844 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:55:19,814 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/B in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,814 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/A is initiating minor compaction (all files) 2023-04-01 14:55:19,814 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/166931bac05540a89868ec939a57aea0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/0eac0a24f1a946d799b8a94b10ec0701, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2dd8268f77274b09ba85f0b79cf62e94, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a1a9faa70b114fbb8de847113aa8faa2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/5615cff9c4134ea8a29fe797d4622086] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=59.2 K 2023-04-01 14:55:19,814 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/A in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,814 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/02e43b49d272474b90b86a7a9ca3dd4c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9312c3bfd43745de8db6530804a9c0d6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/5e1ef2fc8cec431c87bf46e37ec9a988, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6ddf3c6321d24f6db4f1b7bfb70f5db9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/44daaa7b19e4421fa8ca97910031e069] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=161.0 K 2023-04-01 14:55:19,814 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 166931bac05540a89868ec939a57aea0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680360914564 2023-04-01 14:55:19,815 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 02e43b49d272474b90b86a7a9ca3dd4c, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680360914564 2023-04-01 14:55:19,815 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 0eac0a24f1a946d799b8a94b10ec0701, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680360915513 2023-04-01 14:55:19,815 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 9312c3bfd43745de8db6530804a9c0d6, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680360915384 2023-04-01 14:55:19,815 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2dd8268f77274b09ba85f0b79cf62e94, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680360915534 2023-04-01 14:55:19,815 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 5e1ef2fc8cec431c87bf46e37ec9a988, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680360915534 2023-04-01 14:55:19,815 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a1a9faa70b114fbb8de847113aa8faa2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680360915881 2023-04-01 14:55:19,815 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 6ddf3c6321d24f6db4f1b7bfb70f5db9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680360915881 2023-04-01 14:55:19,816 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 5615cff9c4134ea8a29fe797d4622086, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680360918045 2023-04-01 14:55:19,816 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 44daaa7b19e4421fa8ca97910031e069, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680360918035 2023-04-01 14:55:19,825 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#B#compaction#345 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:19,828 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#A#compaction#346 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:19,837 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:19,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:19,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:19,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:19,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:19,840 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:19,840 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,840 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:19,840 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:19,841 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,841 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,842 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:19,842 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:19,842 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:19,842 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:19,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:19,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:19,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:19,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:19,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:19,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:19,844 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:19,844 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:19,844 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:19,844 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:19,844 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:55:19,844 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:19,844 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:19,844 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:19,844 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:19,844 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:19,844 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:20,234 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/d0d434a13ddf479bb596cc72f31b09f7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/d0d434a13ddf479bb596cc72f31b09f7 2023-04-01 14:55:20,238 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/B of b2dc695f920ae42c5964c2b96bd0ec47 into d0d434a13ddf479bb596cc72f31b09f7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:20,238 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:20,238 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/B, priority=11, startTime=1680360919812; duration=0sec 2023-04-01 14:55:20,238 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:20,238 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:55:20,243 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60590 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:55:20,243 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/C is initiating minor compaction (all files) 2023-04-01 14:55:20,243 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/C in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:20,243 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/3c6294c262534eb58262d8def66d4281, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/483fe001c759453b948fa0a8451cbba0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c7848e140bf24b46ab755fb8933d29ba, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9b5f76bd4dc44149836f2fc753e42a1f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9058932546d14a20978fc9308a0eb8f5] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=59.2 K 2023-04-01 14:55:20,244 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3c6294c262534eb58262d8def66d4281, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680360914564 2023-04-01 14:55:20,244 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 483fe001c759453b948fa0a8451cbba0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680360915513 2023-04-01 14:55:20,245 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c7848e140bf24b46ab755fb8933d29ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1680360915534 2023-04-01 14:55:20,245 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9b5f76bd4dc44149836f2fc753e42a1f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680360915881 2023-04-01 14:55:20,246 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9058932546d14a20978fc9308a0eb8f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680360918045 2023-04-01 14:55:20,264 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#C#compaction#348 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:20,266 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401f50c6f18a9734e1d8b41ab7f4f89f909 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f50c6f18a9734e1d8b41ab7f4f89f909 2023-04-01 14:55:20,266 WARN [DataStreamer for file /user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/70824fbff9a7420894c3ac88c604847c] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-01 14:55:20,283 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/70824fbff9a7420894c3ac88c604847c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/70824fbff9a7420894c3ac88c604847c 2023-04-01 14:55:20,290 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/A of b2dc695f920ae42c5964c2b96bd0ec47 into 70824fbff9a7420894c3ac88c604847c(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:20,290 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:20,290 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/A, priority=11, startTime=1680360919812; duration=0sec 2023-04-01 14:55:20,290 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:20,303 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=246, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/43167892b82f4ec4bc9b809b99986cf0 2023-04-01 14:55:20,310 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/9e1a53cc40b54b04bd52eaa59ec914e1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9e1a53cc40b54b04bd52eaa59ec914e1 2023-04-01 14:55:20,320 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/C of b2dc695f920ae42c5964c2b96bd0ec47 into 9e1a53cc40b54b04bd52eaa59ec914e1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:20,320 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:20,320 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/C, priority=11, startTime=1680360919813; duration=0sec 2023-04-01 14:55:20,320 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:20,324 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/b5d8bb6fcf84474589c26a30b7f2c7ea 2023-04-01 14:55:20,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:20,325 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:20,346 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/b0d19d8ceb6942ee9ea9555b48d567f3 2023-04-01 14:55:20,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360980341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360980342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360980342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360980347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,350 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/43167892b82f4ec4bc9b809b99986cf0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/43167892b82f4ec4bc9b809b99986cf0 2023-04-01 14:55:20,353 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/43167892b82f4ec4bc9b809b99986cf0, entries=150, sequenceid=246, filesize=25.3 K 2023-04-01 14:55:20,353 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/b5d8bb6fcf84474589c26a30b7f2c7ea as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b5d8bb6fcf84474589c26a30b7f2c7ea 2023-04-01 14:55:20,358 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b5d8bb6fcf84474589c26a30b7f2c7ea, entries=150, sequenceid=246, filesize=11.8 K 2023-04-01 14:55:20,359 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/b0d19d8ceb6942ee9ea9555b48d567f3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/b0d19d8ceb6942ee9ea9555b48d567f3 2023-04-01 14:55:20,362 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/b0d19d8ceb6942ee9ea9555b48d567f3, entries=150, sequenceid=246, filesize=11.8 K 2023-04-01 14:55:20,362 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for b2dc695f920ae42c5964c2b96bd0ec47 in 518ms, sequenceid=246, compaction requested=false 2023-04-01 14:55:20,362 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:20,362 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:20,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:20,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:20,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:20,363 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:20,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,365 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,365 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,365 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:20,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c8f61[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:20,367 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:20,367 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,368 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,368 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:20,368 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:20,368 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:20,368 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,368 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:20,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:20,369 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:20,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:20,372 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:20,372 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:20,372 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:20,372 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:20,372 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:20,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,374 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,378 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:20,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:20,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:20,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:20,383 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:20,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:20,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,384 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,384 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,384 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:20,383 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:20,384 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:20,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:20,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:20,453 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:20,454 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:20,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:20,454 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:20,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:20,454 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:20,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:20,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360980471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360980472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360980472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360980478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,484 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:20,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:20,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:20,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:20,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:20,486 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:20,486 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:20,486 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:20,486 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:20,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,488 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:20,488 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:20,488 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,488 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:20,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:20,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:20,490 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:20,490 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:20,490 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:20,490 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:20,490 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:20,490 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:20,490 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,490 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:20,490 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:20,491 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,491 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:20,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b20dfaa[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:20,493 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:20,493 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,494 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:20,494 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,494 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:20,496 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,496 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:20,496 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,496 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:20,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:20,500 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:20,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:20,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:20,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:20,500 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:20,501 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,501 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,501 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:20,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,502 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,503 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:20,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:20,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:20,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:20,506 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,506 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,506 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,506 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,506 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,506 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:20,506 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:20,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,506 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,507 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:20,507 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:20,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360980580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360980581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360980581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360980581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,607 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:20,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:20,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:20,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:20,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:20,610 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:20,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:20,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:20,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:20,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:20,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:20,611 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,612 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,612 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:20,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:20,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:20,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:20,613 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:20,613 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:20,613 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:20,618 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:20,618 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:20,618 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:20,618 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:20,619 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:20,619 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:20,621 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,621 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:20,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c620494[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:20,623 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:20,623 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,624 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,624 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:20,624 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:20,624 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:20,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:20,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:20,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:20,628 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:20,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:20,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:20,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:20,628 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:20,629 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,629 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,630 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:20,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,630 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,631 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,634 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:20,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:20,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:20,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:20,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:20,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:20,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:20,634 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:20,634 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:20,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:20,734 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:20,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:20,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:20,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:20,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:20,738 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:20,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:20,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:20,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:20,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,739 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:20,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:20,739 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:20,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:20,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:20,741 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:20,741 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:20,741 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:20,741 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:20,742 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,742 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c6d4052[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:20,744 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:20,744 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,746 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,746 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:20,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:20,746 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:20,746 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:20,748 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:20,750 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:20,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:20,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:20,750 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:20,750 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:20,751 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,751 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,752 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,754 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:20,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:20,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:20,759 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:20,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:20,759 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:20,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:20,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:20,759 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:20,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:20,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:20,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360980787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360980787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360980788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:20,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360980788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,859 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:20,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:20,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:20,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:20,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:20,863 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:20,863 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:20,863 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:20,863 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:20,863 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,863 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,865 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,865 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:20,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:20,865 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,865 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:20,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:20,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:20,873 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:20,873 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:20,873 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:20,873 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:20,874 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:20,874 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:20,876 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,876 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38ce8b39[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:20,879 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:20,879 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,880 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,880 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:20,880 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:20,880 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304010d2fd944fe784e2394294cd3e899b8da to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304010d2fd944fe784e2394294cd3e899b8da 2023-04-01 14:55:20,880 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:20,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:20,882 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:20,887 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:20,887 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:20,887 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:20,887 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:20,887 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:20,890 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,890 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:20,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,890 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:20,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,891 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:20,892 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,894 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,894 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,894 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:20,894 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:20,894 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:20,894 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:20,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:20,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:20,895 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:20,894 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:20,895 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:20,895 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:20,895 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:20,895 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:20,995 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:20,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:20,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:20,998 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:20,998 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:20,998 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:20,999 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:20,999 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:20,999 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,000 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,000 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,000 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,000 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:21,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:21,001 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:21,001 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:21,002 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,002 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:21,002 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:21,002 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:21,003 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,003 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65cece05[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:21,005 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:21,005 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,007 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:21,008 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,008 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:21,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:21,011 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:21,011 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:21,011 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:21,011 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:21,011 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:21,013 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,013 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,013 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,013 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,015 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,016 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,016 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,016 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,016 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,016 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,017 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,017 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,017 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,017 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,017 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:21,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:21,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:21,018 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:21,018 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:21,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:21,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360981092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:21,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360981092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:21,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360981092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:21,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360981092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,118 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:21,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:21,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:21,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:21,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:21,121 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:21,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:21,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:21,123 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,123 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,125 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,125 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,126 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,126 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:21,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:21,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:21,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:21,127 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:21,127 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,127 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:21,131 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:21,131 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:21,132 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,132 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:21,132 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:21,132 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:21,133 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,134 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,134 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7994208c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:21,135 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:21,135 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,137 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:21,137 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:21,137 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:21,137 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,137 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:21,138 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:21,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:21,140 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:21,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:21,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:21,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:21,140 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:21,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,148 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,151 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,151 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,151 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,151 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,151 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:21,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:21,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:21,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,151 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,152 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,152 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,152 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,152 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,152 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:21,152 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:21,252 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:21,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:21,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:21,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:21,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:21,255 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:21,255 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,256 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:21,256 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:21,256 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,256 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,259 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,259 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,261 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,261 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:21,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:21,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:21,261 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,262 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:21,262 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:21,262 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,262 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:21,262 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:21,262 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,262 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:21,262 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:21,264 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,264 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,264 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,264 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:21,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fcd903c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:21,268 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:21,268 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,269 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,270 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:21,270 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:21,270 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,270 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:21,270 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:21,272 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:21,272 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,272 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:21,272 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:21,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:21,278 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:21,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:21,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:21,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:21,278 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:21,280 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,280 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,280 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,281 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,284 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,288 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=265, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/d87242cbea7d412fa336ec9159eb4087 2023-04-01 14:55:21,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,291 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:21,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:21,291 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:21,291 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:21,292 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,292 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,392 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:21,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:21,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:21,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:21,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:21,395 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:21,395 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,395 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:21,395 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:21,395 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,395 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,398 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:21,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:21,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:21,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:21,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:21,400 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,400 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:21,400 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:21,400 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:21,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:21,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:21,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:21,401 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,401 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,402 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,402 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:21,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3964dbf[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:21,404 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:21,404 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,405 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:21,405 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,405 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:21,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,407 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,407 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,407 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:21,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:21,409 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:21,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:21,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:21,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:21,410 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:21,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,411 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:21,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:21,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,416 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:21,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,416 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:21,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,517 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:21,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:21,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:21,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:21,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:21,520 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:21,520 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,520 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:21,520 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:21,520 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,520 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,521 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,522 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,522 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:21,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:21,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:21,523 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,523 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:21,523 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:21,523 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:21,530 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,530 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:21,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@103b052e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:21,532 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:21,532 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,533 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:21,534 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:21,533 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,534 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:21,534 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:21,535 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,535 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,535 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,535 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:21,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:21,537 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:21,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:21,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:21,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:21,538 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:21,539 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,539 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,539 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,540 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,541 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,544 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,544 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,544 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,544 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:21,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:21,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:21,544 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,545 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,545 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,545 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,545 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:21,545 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:21,545 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,545 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:21,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360981596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:21,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360981597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:21,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360981597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:21,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360981598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,645 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:21,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:21,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:21,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:21,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:21,648 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:21,648 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,648 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:21,648 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:21,648 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,649 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,650 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,650 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,650 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,650 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:21,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:21,652 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:21,652 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,652 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:21,652 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:21,652 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,652 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:21,652 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:21,652 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,652 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:21,652 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:21,656 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4873c7e5[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:21,658 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:21,658 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,660 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,660 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,660 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:21,662 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:21,662 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,662 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,662 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:21,664 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:21,664 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:21,664 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:21,664 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:21,664 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:21,666 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,666 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,666 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,666 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,667 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,669 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:21,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,670 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,670 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:21,670 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:21,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/2ad605a4d95e4516ab68a88fe1ba01d9 2023-04-01 14:55:21,770 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:21,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:21,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:21,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:21,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:21,774 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:21,774 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,774 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:21,774 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:21,774 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,774 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,777 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,777 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,777 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,777 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:21,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:21,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:21,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,778 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:21,786 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,786 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:21,786 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:21,786 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:21,786 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,788 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b1cf2a7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:21,790 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:21,790 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,791 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,791 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:21,791 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:21,791 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:21,791 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,791 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:21,792 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:21,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:21,796 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:21,796 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:21,796 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:21,796 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:21,796 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:21,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,799 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,800 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,803 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,804 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,804 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:21,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:21,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:21,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,804 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:21,804 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:21,905 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:21,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:21,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:21,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:21,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:21,910 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:21,910 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,910 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:21,910 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:21,911 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,911 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,912 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,913 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,913 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:21,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:21,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:21,914 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:21,914 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:21,914 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:21,919 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,919 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:21,919 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:21,919 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:21,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad6b1ce[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:21,922 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:21,922 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,925 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:21,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:21,927 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:21,929 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:21,929 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:21,929 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:21,929 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:21,929 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:21,930 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,930 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:21,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,931 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:21,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:21,931 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,933 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:21,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,935 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:21,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:21,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:21,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:21,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:21,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:21,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:21,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:21,935 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:21,935 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:21,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:21,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:22,035 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:22,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:22,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:22,039 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:22,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:22,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:22,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:22,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,042 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:22,042 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:22,042 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,042 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:22,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:22,043 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:22,043 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:22,044 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:22,044 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:22,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:22,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:22,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:22,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:22,045 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:22,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128cddcf[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:22,047 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,048 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:22,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:22,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:22,048 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:22,052 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:22,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:22,055 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:22,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:22,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:22,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:22,055 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:22,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,056 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:22,057 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,058 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:22,060 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:22,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:22,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:22,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:22,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:22,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:22,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,060 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:22,060 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:22,060 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:22,111 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/f5f7e81dafb8475caf645d589ec4ccc6 2023-04-01 14:55:22,115 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/d87242cbea7d412fa336ec9159eb4087 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d87242cbea7d412fa336ec9159eb4087 2023-04-01 14:55:22,118 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d87242cbea7d412fa336ec9159eb4087, entries=300, sequenceid=265, filesize=46.1 K 2023-04-01 14:55:22,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/2ad605a4d95e4516ab68a88fe1ba01d9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2ad605a4d95e4516ab68a88fe1ba01d9 2023-04-01 14:55:22,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2ad605a4d95e4516ab68a88fe1ba01d9, entries=150, sequenceid=265, filesize=11.9 K 2023-04-01 14:55:22,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/f5f7e81dafb8475caf645d589ec4ccc6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/f5f7e81dafb8475caf645d589ec4ccc6 2023-04-01 14:55:22,133 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/f5f7e81dafb8475caf645d589ec4ccc6, entries=150, sequenceid=265, filesize=11.9 K 2023-04-01 14:55:22,134 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b2dc695f920ae42c5964c2b96bd0ec47 in 1681ms, sequenceid=265, compaction requested=true 2023-04-01 14:55:22,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:22,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:22,134 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:22,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:22,134 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:22,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:22,135 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 99542 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:22,135 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36838 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:22,135 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/A is initiating minor compaction (all files) 2023-04-01 14:55:22,135 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/B is initiating minor compaction (all files) 2023-04-01 14:55:22,135 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/A in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:22,135 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/B in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:22,135 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/70824fbff9a7420894c3ac88c604847c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/43167892b82f4ec4bc9b809b99986cf0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d87242cbea7d412fa336ec9159eb4087] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=97.2 K 2023-04-01 14:55:22,135 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/d0d434a13ddf479bb596cc72f31b09f7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b5d8bb6fcf84474589c26a30b7f2c7ea, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2ad605a4d95e4516ab68a88fe1ba01d9] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=36.0 K 2023-04-01 14:55:22,136 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 70824fbff9a7420894c3ac88c604847c, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680360918045 2023-04-01 14:55:22,136 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting d0d434a13ddf479bb596cc72f31b09f7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680360918045 2023-04-01 14:55:22,136 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 43167892b82f4ec4bc9b809b99986cf0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680360918180 2023-04-01 14:55:22,136 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b5d8bb6fcf84474589c26a30b7f2c7ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680360918180 2023-04-01 14:55:22,137 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d87242cbea7d412fa336ec9159eb4087, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680360920340 2023-04-01 14:55:22,137 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2ad605a4d95e4516ab68a88fe1ba01d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680360920346 2023-04-01 14:55:22,144 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#B#compaction#354 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:22,161 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:22,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:22,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:22,172 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#A#compaction#355 average throughput is 0.70 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:22,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:22,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:22,176 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:22,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:22,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:22,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:22,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,176 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,177 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:22,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:22,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:22,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:22,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:22,179 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:22,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:22,180 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 14:55:22,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:22,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:22,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:22,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:22,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:22,180 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:22,575 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/37e11e9b516942d298ebbdff25e10d40 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/37e11e9b516942d298ebbdff25e10d40 2023-04-01 14:55:22,579 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/B of b2dc695f920ae42c5964c2b96bd0ec47 into 37e11e9b516942d298ebbdff25e10d40(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:22,579 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:22,579 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/B, priority=13, startTime=1680360922134; duration=0sec 2023-04-01 14:55:22,579 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:22,579 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:22,580 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36838 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:22,580 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/C is initiating minor compaction (all files) 2023-04-01 14:55:22,580 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/C in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:22,580 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9e1a53cc40b54b04bd52eaa59ec914e1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/b0d19d8ceb6942ee9ea9555b48d567f3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/f5f7e81dafb8475caf645d589ec4ccc6] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=36.0 K 2023-04-01 14:55:22,580 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9e1a53cc40b54b04bd52eaa59ec914e1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680360918045 2023-04-01 14:55:22,581 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b0d19d8ceb6942ee9ea9555b48d567f3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680360918180 2023-04-01 14:55:22,581 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f5f7e81dafb8475caf645d589ec4ccc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680360920346 2023-04-01 14:55:22,587 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#C#compaction#357 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:22,593 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401ba8b568ac46541e2b1593c5f6f787fcc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ba8b568ac46541e2b1593c5f6f787fcc 2023-04-01 14:55:22,595 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/ac3645fe5eff423baf5e84fce14a2391 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ac3645fe5eff423baf5e84fce14a2391 2023-04-01 14:55:22,598 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=281, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/6fb8950d2f2d4869964df842a72492c2 2023-04-01 14:55:22,600 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/C of b2dc695f920ae42c5964c2b96bd0ec47 into ac3645fe5eff423baf5e84fce14a2391(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:22,600 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:22,600 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/C, priority=13, startTime=1680360922134; duration=0sec 2023-04-01 14:55:22,600 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:22,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:22,607 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:22,609 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/6538b64854814f7999790f8cdd2ed18d 2023-04-01 14:55:22,617 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/9373bbf3502a45b18f615694e2d444e3 2023-04-01 14:55:22,620 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/6fb8950d2f2d4869964df842a72492c2 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6fb8950d2f2d4869964df842a72492c2 2023-04-01 14:55:22,624 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6fb8950d2f2d4869964df842a72492c2, entries=150, sequenceid=281, filesize=25.4 K 2023-04-01 14:55:22,625 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/6538b64854814f7999790f8cdd2ed18d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/6538b64854814f7999790f8cdd2ed18d 2023-04-01 14:55:22,628 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/6538b64854814f7999790f8cdd2ed18d, entries=150, sequenceid=281, filesize=11.9 K 2023-04-01 14:55:22,629 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/9373bbf3502a45b18f615694e2d444e3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9373bbf3502a45b18f615694e2d444e3 2023-04-01 14:55:22,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360982623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360982624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,633 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9373bbf3502a45b18f615694e2d444e3, entries=150, sequenceid=281, filesize=11.9 K 2023-04-01 14:55:22,635 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for b2dc695f920ae42c5964c2b96bd0ec47 in 455ms, sequenceid=281, compaction requested=false 2023-04-01 14:55:22,635 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:22,635 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:22,635 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:22,635 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:22,635 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,635 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:22,635 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:22,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:22,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:55:22,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:22,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:22,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:22,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:22,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:22,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:22,637 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,637 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,638 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,638 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44167c94[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:22,640 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:22,640 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,641 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:22,641 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:22,641 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:22,641 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,641 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:22,642 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:22,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,642 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:22,642 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:22,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:22,653 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:22,653 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:22,654 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:22,654 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:22,654 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:22,655 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,655 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,655 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,655 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:22,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,657 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,659 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304019e3f331af7be4ddc85301ab93afc1637 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019e3f331af7be4ddc85301ab93afc1637 2023-04-01 14:55:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,661 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,661 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,661 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,661 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:22,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:22,662 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,662 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,662 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,662 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:22,662 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:22,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:22,661 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:22,662 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:22,663 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:22,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:22,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:22,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360982678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360982679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360982731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360982731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,762 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:22,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:22,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:22,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:22,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:22,765 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:22,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:22,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:22,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:22,766 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,766 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:22,767 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:22,767 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,767 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:22,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:22,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:22,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:22,768 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:22,768 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:22,769 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:22,769 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:22,769 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:22,769 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:22,769 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,769 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:22,769 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:22,770 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,770 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,770 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,770 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:22,772 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:22,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4813277b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:22,772 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,773 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:22,773 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,773 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:22,775 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:22,775 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,775 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,775 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:22,777 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:22,777 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:22,777 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:22,777 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:22,777 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:22,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:22,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,781 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:22,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:22,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:22,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:22,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:22,783 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:22,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:22,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:22,783 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:22,783 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:22,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:22,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360982786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360982786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,884 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:22,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:22,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:22,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:22,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:22,889 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:22,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:22,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:22,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:22,890 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,890 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,891 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,891 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:22,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:22,892 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,892 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:22,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:22,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:22,893 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:22,893 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:22,893 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:22,893 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:22,896 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,896 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:22,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27a4b776[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:22,897 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:22,897 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,898 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:22,899 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:22,899 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,899 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:22,900 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:22,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:22,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:22,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:22,902 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:22,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:22,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:22,902 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:22,902 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:22,907 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,907 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,907 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:22,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,908 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:22,909 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:22,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:22,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:22,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:22,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:22,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:22,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:22,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:22,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:22,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:22,913 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:22,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:22,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:22,914 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:22,914 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:22,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:22,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360982936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360982937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360982991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:22,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360982992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:22,993 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/33552577916a4b91a81864f9eaeeb87d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/33552577916a4b91a81864f9eaeeb87d 2023-04-01 14:55:22,997 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/A of b2dc695f920ae42c5964c2b96bd0ec47 into 33552577916a4b91a81864f9eaeeb87d(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:22,997 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:22,997 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/A, priority=13, startTime=1680360922134; duration=0sec 2023-04-01 14:55:22,997 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:23,014 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:23,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:23,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:23,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:23,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:23,017 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:23,017 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:23,017 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:23,017 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:23,018 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,018 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,019 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,019 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:23,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:23,019 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,019 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:23,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:23,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:23,021 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:23,021 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:23,021 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:23,021 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:23,021 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:23,021 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,021 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:23,021 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:23,021 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,022 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:23,022 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:23,022 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,022 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:23,022 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:23,024 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,025 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,025 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,025 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:23,028 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:23,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53b264e8[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:23,028 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,029 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,029 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:23,029 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:23,029 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:23,029 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,029 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:23,030 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,030 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:23,030 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,030 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:23,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:23,033 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:23,033 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:23,033 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:23,033 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:23,033 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:23,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:23,034 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:23,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:23,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:23,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:23,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:23,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:23,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:23,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,039 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:23,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,039 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:23,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:23,078 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=301, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/98480c45c71e4a3fae9f34b42ad63bfc 2023-04-01 14:55:23,140 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:23,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:23,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:23,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:23,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:23,143 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:23,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:23,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:23,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:23,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,144 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:23,144 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,145 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,145 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:23,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:23,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:23,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:23,146 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,146 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:23,146 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:23,146 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,147 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:23,147 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:23,148 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:23,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@243750d6[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:23,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:23,150 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,150 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:23,151 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,151 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:23,153 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,153 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,153 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:23,153 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:23,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:23,155 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:23,155 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:23,155 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:23,155 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:23,155 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:23,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,156 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:23,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,158 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,159 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,161 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,161 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,161 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:23,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:23,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:23,161 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:23,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:23,161 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:23,161 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,162 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:23,162 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:23,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:23,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:23,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,162 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:23,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360983242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:23,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360983243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,262 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:23,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:23,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:23,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:23,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:23,265 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:23,265 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:23,265 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:23,265 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:23,266 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,266 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,267 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:23,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:23,267 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,267 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:23,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:23,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:23,269 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,269 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:23,269 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:23,269 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:23,269 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:23,272 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,272 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,272 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,272 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:23,274 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:23,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75ce93d4[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:23,274 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,275 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:23,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:23,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:23,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:23,279 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:23,279 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:23,279 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:23,279 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:23,279 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:23,280 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,280 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,280 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,281 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,282 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,288 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:23,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:23,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:23,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:23,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:23,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:23,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:23,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:23,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,288 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:23,288 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:23,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:23,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:23,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360983293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:23,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360983294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,388 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:23,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:23,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:23,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:23,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:23,396 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:23,396 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:23,396 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:23,396 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:23,397 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,397 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,399 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:23,399 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:23,399 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,399 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:23,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:23,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:23,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:23,401 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,401 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:23,401 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:23,401 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:23,402 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,402 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,403 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,403 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:23,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7753851c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:23,404 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:23,404 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,405 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:23,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:23,406 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,406 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:23,412 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,412 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:23,412 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:23,412 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:23,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:23,417 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:23,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:23,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:23,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:23,417 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:23,419 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,419 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:23,419 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:23,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:23,420 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,421 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:23,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:23,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:23,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:23,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:23,424 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:23,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:23,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:23,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:23,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:23,425 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:23,425 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:23,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/b979e7b8a51f4f488513085645483164 2023-04-01 14:55:23,502 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/d00f5eecd40041c4810c36cbd3e9757f 2023-04-01 14:55:23,505 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/98480c45c71e4a3fae9f34b42ad63bfc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/98480c45c71e4a3fae9f34b42ad63bfc 2023-04-01 14:55:23,508 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/98480c45c71e4a3fae9f34b42ad63bfc, entries=250, sequenceid=301, filesize=39.2 K 2023-04-01 14:55:23,509 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/b979e7b8a51f4f488513085645483164 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b979e7b8a51f4f488513085645483164 2023-04-01 14:55:23,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b979e7b8a51f4f488513085645483164, entries=150, sequenceid=301, filesize=11.9 K 2023-04-01 14:55:23,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/d00f5eecd40041c4810c36cbd3e9757f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d00f5eecd40041c4810c36cbd3e9757f 2023-04-01 14:55:23,515 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d00f5eecd40041c4810c36cbd3e9757f, entries=150, sequenceid=301, filesize=11.9 K 2023-04-01 14:55:23,516 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b2dc695f920ae42c5964c2b96bd0ec47 in 881ms, sequenceid=301, compaction requested=true 2023-04-01 14:55:23,516 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:23,516 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:23,516 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:23,516 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:23,516 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:23,516 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:23,517 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92904 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:23,517 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:23,517 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/A is initiating minor compaction (all files) 2023-04-01 14:55:23,517 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/B is initiating minor compaction (all files) 2023-04-01 14:55:23,517 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/A in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,517 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/B in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,517 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/33552577916a4b91a81864f9eaeeb87d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6fb8950d2f2d4869964df842a72492c2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/98480c45c71e4a3fae9f34b42ad63bfc] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=90.7 K 2023-04-01 14:55:23,517 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/37e11e9b516942d298ebbdff25e10d40, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/6538b64854814f7999790f8cdd2ed18d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b979e7b8a51f4f488513085645483164] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=36.4 K 2023-04-01 14:55:23,519 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 33552577916a4b91a81864f9eaeeb87d, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680360920346 2023-04-01 14:55:23,519 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 37e11e9b516942d298ebbdff25e10d40, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680360920346 2023-04-01 14:55:23,519 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 6fb8950d2f2d4869964df842a72492c2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680360920469 2023-04-01 14:55:23,520 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6538b64854814f7999790f8cdd2ed18d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680360920469 2023-04-01 14:55:23,520 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 98480c45c71e4a3fae9f34b42ad63bfc, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680360922623 2023-04-01 14:55:23,520 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b979e7b8a51f4f488513085645483164, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680360922629 2023-04-01 14:55:23,525 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:23,525 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#B#compaction#364 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:23,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:23,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:23,527 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#A#compaction#363 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:23,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:23,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:23,528 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:23,528 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:23,528 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:23,528 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:23,528 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,529 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:23,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:23,531 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,532 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,532 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:23,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:23,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:23,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:23,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:23,533 WARN [DataStreamer for file /user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/f073554e840747cfb1585f845183a949] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-01 14:55:23,533 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:23,533 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:23,534 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:23,534 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:23,534 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:23,534 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:23,534 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:23,534 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:23,534 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:23,538 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/f073554e840747cfb1585f845183a949 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/f073554e840747cfb1585f845183a949 2023-04-01 14:55:23,542 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/B of b2dc695f920ae42c5964c2b96bd0ec47 into f073554e840747cfb1585f845183a949(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:23,542 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:23,542 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/B, priority=13, startTime=1680360923516; duration=0sec 2023-04-01 14:55:23,542 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:23,542 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:23,543 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:23,543 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/C is initiating minor compaction (all files) 2023-04-01 14:55:23,543 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/C in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:23,543 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ac3645fe5eff423baf5e84fce14a2391, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9373bbf3502a45b18f615694e2d444e3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d00f5eecd40041c4810c36cbd3e9757f] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=36.4 K 2023-04-01 14:55:23,544 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ac3645fe5eff423baf5e84fce14a2391, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680360920346 2023-04-01 14:55:23,544 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9373bbf3502a45b18f615694e2d444e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680360920469 2023-04-01 14:55:23,545 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting d00f5eecd40041c4810c36cbd3e9757f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680360922629 2023-04-01 14:55:23,548 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401c20402fd6bff4e0eac3ebfa064fbaaa6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c20402fd6bff4e0eac3ebfa064fbaaa6 2023-04-01 14:55:23,549 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/3850f0a17f214b94bb3035bc9ffb9504 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/3850f0a17f214b94bb3035bc9ffb9504 2023-04-01 14:55:23,551 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#C#compaction#366 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:23,552 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/A of b2dc695f920ae42c5964c2b96bd0ec47 into 3850f0a17f214b94bb3035bc9ffb9504(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:23,552 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:23,553 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/A, priority=13, startTime=1680360923516; duration=0sec 2023-04-01 14:55:23,553 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:23,557 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=316, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/9ac50d8a297e4214ad99afbd6409fd2a 2023-04-01 14:55:23,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:23,752 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:23,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:23,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360983788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:23,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360983788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:23,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360983797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360983801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:23,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360983893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:23,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360983893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:23,962 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/144da3f3a6f7442187144d3de92dd5b5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/144da3f3a6f7442187144d3de92dd5b5 2023-04-01 14:55:23,967 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/326200a0ffad4dbdbf1af5674fe7144d 2023-04-01 14:55:23,982 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/C of b2dc695f920ae42c5964c2b96bd0ec47 into 144da3f3a6f7442187144d3de92dd5b5(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:23,982 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:23,982 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/C, priority=13, startTime=1680360923516; duration=0sec 2023-04-01 14:55:23,982 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:24,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:24,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43492 deadline: 1680360984099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:24,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360984100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:24,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360984404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:24,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43480 deadline: 1680360984405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,410 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/14d873cf3a09417f949c4755f9ff1364 2023-04-01 14:55:24,414 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/9ac50d8a297e4214ad99afbd6409fd2a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9ac50d8a297e4214ad99afbd6409fd2a 2023-04-01 14:55:24,418 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9ac50d8a297e4214ad99afbd6409fd2a, entries=150, sequenceid=316, filesize=25.4 K 2023-04-01 14:55:24,418 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/326200a0ffad4dbdbf1af5674fe7144d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/326200a0ffad4dbdbf1af5674fe7144d 2023-04-01 14:55:24,422 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/326200a0ffad4dbdbf1af5674fe7144d, entries=150, sequenceid=316, filesize=11.9 K 2023-04-01 14:55:24,422 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/14d873cf3a09417f949c4755f9ff1364 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/14d873cf3a09417f949c4755f9ff1364 2023-04-01 14:55:24,425 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/14d873cf3a09417f949c4755f9ff1364, entries=150, sequenceid=316, filesize=11.9 K 2023-04-01 14:55:24,426 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for b2dc695f920ae42c5964c2b96bd0ec47 in 892ms, sequenceid=316, compaction requested=false 2023-04-01 14:55:24,426 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:24,426 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:24,426 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:24,429 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:24,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37862a3a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:24,430 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:24,430 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:24,432 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:24,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:24,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:24,432 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,432 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:24,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:24,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:24,433 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:24,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:24,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:24,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:24,436 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:24,436 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:24,436 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:24,436 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:24,436 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:24,437 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:24,437 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:24,437 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:24,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:24,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:24,438 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,439 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:24,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:24,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:24,442 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:24,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:24,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:24,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:24,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:24,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:24,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:24,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:24,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:24,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:24,443 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:24,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:24,443 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:24,543 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:24,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:24,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:24,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:24,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:24,546 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:24,546 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:24,546 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:24,546 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:24,547 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:24,547 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:24,548 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:24,548 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:24,548 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,548 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:24,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:24,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:24,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:24,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:24,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:24,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:24,550 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:24,550 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:24,550 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:55:24,550 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:24,550 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:24,550 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:24,550 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:24,550 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:24,550 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:24,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:24,812 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:24,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:24,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360984828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:24,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43514 deadline: 1680360984829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:24,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360984910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:24,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360984912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:24,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360984936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:24,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360984936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:24,963 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401c11ceac00a9046fe99cf96772146dea2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c11ceac00a9046fe99cf96772146dea2 2023-04-01 14:55:24,974 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=338, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/a2ae2763003d45e6bb33c1135864e66b 2023-04-01 14:55:25,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:25,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360985142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:25,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360985143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,390 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/1db25a37137d451a84d3a73fed5d6c44 2023-04-01 14:55:25,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:25,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360985448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:25,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360985449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,802 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/405a31347cde4bb0974af5c429f10dfb 2023-04-01 14:55:25,806 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/a2ae2763003d45e6bb33c1135864e66b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/a2ae2763003d45e6bb33c1135864e66b 2023-04-01 14:55:25,809 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/a2ae2763003d45e6bb33c1135864e66b, entries=150, sequenceid=338, filesize=25.4 K 2023-04-01 14:55:25,810 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/1db25a37137d451a84d3a73fed5d6c44 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1db25a37137d451a84d3a73fed5d6c44 2023-04-01 14:55:25,813 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1db25a37137d451a84d3a73fed5d6c44, entries=150, sequenceid=338, filesize=11.9 K 2023-04-01 14:55:25,813 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/405a31347cde4bb0974af5c429f10dfb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/405a31347cde4bb0974af5c429f10dfb 2023-04-01 14:55:25,816 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/405a31347cde4bb0974af5c429f10dfb, entries=150, sequenceid=338, filesize=11.9 K 2023-04-01 14:55:25,817 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for b2dc695f920ae42c5964c2b96bd0ec47 in 1267ms, sequenceid=338, compaction requested=true 2023-04-01 14:55:25,817 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:25,817 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:25,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:25,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:25,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:25,817 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:25,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,819 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:25,819 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,819 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:25,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:25,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29296dfc[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:25,820 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:25,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,821 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,822 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:25,822 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:25,822 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:25,822 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,822 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:25,824 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:25,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,824 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:25,824 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:25,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:25,827 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:25,827 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:25,827 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:25,827 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:25,827 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:25,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:25,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:25,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,829 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,830 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:25,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:25,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:25,833 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:25,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:25,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:25,833 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:25,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:25,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:25,833 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:25,833 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:25,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:25,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:25,925 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:25,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:25,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:25,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:25,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:25,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:25,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:25,933 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:25,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:25,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:25,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:25,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:25,937 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:25,937 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:25,937 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:25,937 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:25,937 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,937 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:25,943 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:25,943 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:25,943 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,943 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:25,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:25,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:25,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:25,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:25,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:25,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:25,945 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:25,945 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:25,945 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:25,945 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:25,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:25,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:25,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:25,945 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:25,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,946 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:25,947 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,947 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28781b3e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:25,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:25,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,949 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,949 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:25,949 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:25,949 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:25,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,950 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:25,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:25,952 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:25,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:25,954 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:25,954 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:25,954 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:25,954 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:25,954 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:25,956 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,956 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:25,956 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,957 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:25,958 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:25,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:25,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:25,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:25,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:25,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:25,960 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:25,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:25,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:25,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:25,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:25,960 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:25,961 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:25,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:25,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360985961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:25,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360985969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:25,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360985970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:25,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:25,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360985970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,061 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:26,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:26,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:26,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:26,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:26,064 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:26,064 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,064 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:26,064 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:26,064 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,064 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,065 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,066 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,066 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:26,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:26,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:26,067 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,067 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:26,067 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:26,068 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:26,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,070 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:26,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:26,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43681048[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:26,073 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:26,073 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,074 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360986071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:26,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,074 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,074 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:26,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360986072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,076 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,076 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,076 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,076 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:26,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:26,079 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:26,079 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:26,079 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:26,079 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:26,079 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:26,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360986077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360986077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,081 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,081 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,081 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:26,081 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,083 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,085 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,086 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,086 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,086 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:26,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:26,086 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,086 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,086 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:26,086 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:26,086 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:26,086 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,086 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:26,086 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:26,186 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:26,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:26,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:26,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:26,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:26,189 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:26,189 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,189 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:26,189 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:26,190 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,190 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,191 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:26,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:26,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:26,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:26,193 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,193 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:26,193 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:26,193 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:26,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:26,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:26,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:26,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,194 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,194 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,194 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:26,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453925e5[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:26,196 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:26,196 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,197 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:26,197 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,197 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:26,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:26,199 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:26,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:26,201 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:26,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:26,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:26,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:26,202 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:26,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,204 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,205 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,207 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,207 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:26,207 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,207 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:26,207 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:26,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:26,207 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,207 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,207 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,207 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,207 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:26,207 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:26,207 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:26,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360986275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360986275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360986280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360986281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,307 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:26,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:26,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:26,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:26,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:26,310 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:26,310 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,310 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:26,310 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:26,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,312 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,312 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,312 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,312 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:26,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:26,313 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:26,313 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,313 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:26,314 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,314 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:26,314 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:26,314 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:26,314 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:26,321 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,321 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,321 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,321 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:26,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@279de633[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:26,324 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:26,324 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,325 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,325 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:26,325 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:26,325 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:26,325 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,325 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,326 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:26,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:26,331 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:26,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:26,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:26,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:26,331 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:26,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:26,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,335 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,336 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:26,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,339 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:26,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:26,340 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401a93eb5ea24054b7fa7054c904ea86b65 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a93eb5ea24054b7fa7054c904ea86b65 2023-04-01 14:55:26,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:26,340 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:26,340 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:26,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:26,345 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=352, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/b856ddd4e12f4f4dabd40b813039a00f 2023-04-01 14:55:26,440 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:26,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:26,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:26,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:26,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:26,447 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:26,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:26,447 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:26,448 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,448 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,449 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:26,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:26,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:26,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:26,450 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:26,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:26,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:26,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:26,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:26,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:26,451 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:26,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:26,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:26,451 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:26,452 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:26,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f54b815[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:26,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:26,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:26,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:26,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:26,457 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,457 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:26,458 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:26,458 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,458 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,458 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:26,461 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:26,461 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:26,462 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:26,462 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:26,462 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:26,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,463 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:26,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,464 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:26,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:26,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:26,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:26,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:26,467 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:26,468 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:26,568 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:26,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:26,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:26,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:26,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:26,571 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:26,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:26,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:26,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,572 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,573 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,573 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,573 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,573 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:26,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:26,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:26,574 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:26,574 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,574 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:26,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:26,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:26,575 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,575 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:26,575 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:26,575 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:26,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:26,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:26,575 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:26,576 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,576 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,577 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:26,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3d1be7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:26,578 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:26,578 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,580 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:26,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:26,580 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:26,580 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,580 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:26,581 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:26,582 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,581 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,582 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:26,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360986580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:26,584 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:26,584 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:26,584 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:26,584 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:26,584 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:26,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360986582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360986583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,589 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,590 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,590 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:26,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:26,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:26,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:26,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:26,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:26,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,591 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:26,591 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:26,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:26,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360986586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,592 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,691 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:26,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:26,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:26,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:26,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:26,695 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:26,695 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,695 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:26,695 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:26,696 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,696 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:26,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:26,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:26,699 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:26,699 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,699 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:26,699 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:26,700 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,700 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:26,700 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:26,700 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:26,702 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:26,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a70988[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:26,704 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:26,704 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:26,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:26,705 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,705 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:26,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:26,709 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:26,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:26,712 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:26,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:26,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:26,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:26,712 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:26,714 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,714 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,714 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:26,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:26,714 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,716 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:26,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:26,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:26,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:26,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:26,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:26,718 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:26,718 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:26,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:26,754 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/1296417c33e142118a3b079149bd5d96 2023-04-01 14:55:26,763 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/d1f9e24b30f8408db6d85c633f6acda8 2023-04-01 14:55:26,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/b856ddd4e12f4f4dabd40b813039a00f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/b856ddd4e12f4f4dabd40b813039a00f 2023-04-01 14:55:26,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/b856ddd4e12f4f4dabd40b813039a00f, entries=200, sequenceid=352, filesize=32.3 K 2023-04-01 14:55:26,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/1296417c33e142118a3b079149bd5d96 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1296417c33e142118a3b079149bd5d96 2023-04-01 14:55:26,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1296417c33e142118a3b079149bd5d96, entries=150, sequenceid=352, filesize=11.9 K 2023-04-01 14:55:26,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/d1f9e24b30f8408db6d85c633f6acda8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d1f9e24b30f8408db6d85c633f6acda8 2023-04-01 14:55:26,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d1f9e24b30f8408db6d85c633f6acda8, entries=150, sequenceid=352, filesize=11.9 K 2023-04-01 14:55:26,779 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b2dc695f920ae42c5964c2b96bd0ec47 in 854ms, sequenceid=352, compaction requested=true 2023-04-01 14:55:26,779 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:26,779 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:26,779 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:26,779 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:55:26,779 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:55:26,779 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:26,780 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112008 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:55:26,780 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:55:26,780 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/A is initiating minor compaction (all files) 2023-04-01 14:55:26,780 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/B is initiating minor compaction (all files) 2023-04-01 14:55:26,780 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/A in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,780 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/B in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,780 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/3850f0a17f214b94bb3035bc9ffb9504, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9ac50d8a297e4214ad99afbd6409fd2a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/a2ae2763003d45e6bb33c1135864e66b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/b856ddd4e12f4f4dabd40b813039a00f] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=109.4 K 2023-04-01 14:55:26,780 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/f073554e840747cfb1585f845183a949, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/326200a0ffad4dbdbf1af5674fe7144d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1db25a37137d451a84d3a73fed5d6c44, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1296417c33e142118a3b079149bd5d96] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=48.4 K 2023-04-01 14:55:26,781 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 3850f0a17f214b94bb3035bc9ffb9504, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680360922629 2023-04-01 14:55:26,781 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f073554e840747cfb1585f845183a949, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680360922629 2023-04-01 14:55:26,781 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 9ac50d8a297e4214ad99afbd6409fd2a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680360922669 2023-04-01 14:55:26,781 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 326200a0ffad4dbdbf1af5674fe7144d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680360922669 2023-04-01 14:55:26,782 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting a2ae2763003d45e6bb33c1135864e66b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680360923777 2023-04-01 14:55:26,782 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1db25a37137d451a84d3a73fed5d6c44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680360923777 2023-04-01 14:55:26,782 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting b856ddd4e12f4f4dabd40b813039a00f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680360924822 2023-04-01 14:55:26,782 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1296417c33e142118a3b079149bd5d96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680360924827 2023-04-01 14:55:26,790 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#B#compaction#375 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:26,793 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#A#compaction#376 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:26,818 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:26,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:26,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:26,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:26,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:26,822 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:26,822 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,822 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:26,822 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:26,822 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,822 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,825 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,825 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:26,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:26,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:26,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:26,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:26,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:26,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:26,827 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:26,827 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:26,827 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:26,827 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:26,827 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:26,827 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:26,827 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:26,827 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:55:26,827 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:26,827 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:26,827 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:26,828 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:26,828 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:26,828 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:27,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:27,094 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:27,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360987104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360987107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360987110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360987111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,201 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/df26be9dfa6847c784c4d8a38f1014e3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/df26be9dfa6847c784c4d8a38f1014e3 2023-04-01 14:55:27,206 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/B of b2dc695f920ae42c5964c2b96bd0ec47 into df26be9dfa6847c784c4d8a38f1014e3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:27,206 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:27,206 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/B, priority=12, startTime=1680360926779; duration=0sec 2023-04-01 14:55:27,206 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:27,206 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:55:27,207 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:55:27,207 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/C is initiating minor compaction (all files) 2023-04-01 14:55:27,207 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/C in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:27,207 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/144da3f3a6f7442187144d3de92dd5b5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/14d873cf3a09417f949c4755f9ff1364, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/405a31347cde4bb0974af5c429f10dfb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d1f9e24b30f8408db6d85c633f6acda8] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=48.4 K 2023-04-01 14:55:27,208 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 144da3f3a6f7442187144d3de92dd5b5, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1680360922629 2023-04-01 14:55:27,208 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 14d873cf3a09417f949c4755f9ff1364, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680360922669 2023-04-01 14:55:27,209 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 405a31347cde4bb0974af5c429f10dfb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680360923777 2023-04-01 14:55:27,209 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting d1f9e24b30f8408db6d85c633f6acda8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680360924827 2023-04-01 14:55:27,217 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#C#compaction#378 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:27,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360987212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360987214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360987219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360987219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,221 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/7410269f09f04812815b0ba2ec72b7fc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7410269f09f04812815b0ba2ec72b7fc 2023-04-01 14:55:27,225 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/A of b2dc695f920ae42c5964c2b96bd0ec47 into 7410269f09f04812815b0ba2ec72b7fc(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:27,226 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:27,226 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/A, priority=12, startTime=1680360926779; duration=0sec 2023-04-01 14:55:27,226 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:27,230 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/6780f0d5d7a844a4bbf363d758a5530e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/6780f0d5d7a844a4bbf363d758a5530e 2023-04-01 14:55:27,233 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/C of b2dc695f920ae42c5964c2b96bd0ec47 into 6780f0d5d7a844a4bbf363d758a5530e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:27,233 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:27,233 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/C, priority=12, startTime=1680360926779; duration=0sec 2023-04-01 14:55:27,233 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:27,265 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401149596dae2db454a832d421fef7f9ed1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401149596dae2db454a832d421fef7f9ed1 2023-04-01 14:55:27,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360987420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360987422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360987422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360987422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,680 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=370, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/e33e94157a8242df8ead02fa9102f4a0 2023-04-01 14:55:27,698 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/bc24939b25a24660be8bfcfb0a05907a 2023-04-01 14:55:27,714 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/56530be916ba489fbd22012907c560a7 2023-04-01 14:55:27,723 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/e33e94157a8242df8ead02fa9102f4a0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/e33e94157a8242df8ead02fa9102f4a0 2023-04-01 14:55:27,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360987724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,729 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/e33e94157a8242df8ead02fa9102f4a0, entries=150, sequenceid=370, filesize=25.4 K 2023-04-01 14:55:27,729 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-01 14:55:27,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360987727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360987727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,730 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/bc24939b25a24660be8bfcfb0a05907a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/bc24939b25a24660be8bfcfb0a05907a 2023-04-01 14:55:27,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:27,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360987728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,734 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/bc24939b25a24660be8bfcfb0a05907a, entries=150, sequenceid=370, filesize=11.9 K 2023-04-01 14:55:27,735 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/56530be916ba489fbd22012907c560a7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/56530be916ba489fbd22012907c560a7 2023-04-01 14:55:27,740 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/56530be916ba489fbd22012907c560a7, entries=150, sequenceid=370, filesize=11.9 K 2023-04-01 14:55:27,741 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b2dc695f920ae42c5964c2b96bd0ec47 in 914ms, sequenceid=370, compaction requested=false 2023-04-01 14:55:27,741 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:27,741 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:27,743 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:27,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:27,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@843c3d1[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:27,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:27,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:27,747 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:27,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:27,747 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:27,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,747 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:27,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:27,747 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:27,750 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:27,750 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:27,750 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:27,751 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:27,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:27,755 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:27,755 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:27,755 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:27,755 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:27,755 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:27,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:27,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:27,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:27,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:27,758 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:27,760 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:27,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:27,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:27,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:27,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:27,763 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:27,763 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:27,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:27,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:27,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:27,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:27,763 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:27,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:27,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:27,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:27,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:27,863 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:27,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:27,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:27,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:27,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:27,866 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:27,866 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:27,867 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:27,867 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:27,867 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:27,867 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:27,868 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:27,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:27,868 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:27,868 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:27,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:27,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:27,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:27,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:27,870 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:27,870 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:27,870 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:27,870 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:27,870 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:27,870 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:27,870 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:27,870 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:27,870 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:27,870 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:27,871 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:27,871 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:27,871 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:27,871 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:27,885 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304019e49d26506d94d26abd538ad5e9d9fe0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019e49d26506d94d26abd538ad5e9d9fe0 2023-04-01 14:55:27,889 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=387, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/016e33eff6474a369e29ed23911f27aa 2023-04-01 14:55:27,911 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/92e1092bc87f4f01a956b70076f95a12 2023-04-01 14:55:28,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:28,232 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:28,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360988272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360988273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360988274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360988278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,321 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/06cfd8fabd0e4acfb16b2b9ef34a4d72 2023-04-01 14:55:28,325 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/016e33eff6474a369e29ed23911f27aa as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/016e33eff6474a369e29ed23911f27aa 2023-04-01 14:55:28,328 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/016e33eff6474a369e29ed23911f27aa, entries=150, sequenceid=387, filesize=25.4 K 2023-04-01 14:55:28,329 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/92e1092bc87f4f01a956b70076f95a12 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/92e1092bc87f4f01a956b70076f95a12 2023-04-01 14:55:28,340 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/92e1092bc87f4f01a956b70076f95a12, entries=150, sequenceid=387, filesize=11.9 K 2023-04-01 14:55:28,341 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/06cfd8fabd0e4acfb16b2b9ef34a4d72 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/06cfd8fabd0e4acfb16b2b9ef34a4d72 2023-04-01 14:55:28,346 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/06cfd8fabd0e4acfb16b2b9ef34a4d72, entries=150, sequenceid=387, filesize=11.9 K 2023-04-01 14:55:28,346 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b2dc695f920ae42c5964c2b96bd0ec47 in 476ms, sequenceid=387, compaction requested=true 2023-04-01 14:55:28,346 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:28,346 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:28,346 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,348 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:28,348 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:28,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:28,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62bd56dc[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:28,351 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:28,351 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,352 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,352 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:28,352 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:28,352 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:28,352 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,352 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:28,354 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,354 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:28,354 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,354 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:28,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:28,357 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:28,357 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:28,357 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:28,357 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:28,358 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:28,359 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,359 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,359 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,359 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,361 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:28,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,363 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:28,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:28,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:28,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:28,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,363 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:28,363 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:28,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:28,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:28,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:55:28,383 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:28,383 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:28,383 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:28,383 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:28,383 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:28,383 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:28,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360988397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360988398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360988399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360988402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,463 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:28,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:28,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:28,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:28,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:28,466 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:28,466 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:28,466 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:28,466 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:28,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,468 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:28,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:28,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:28,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:28,469 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:28,469 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:28,469 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:28,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:28,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:28,470 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:28,470 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:28,470 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:28,470 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:28,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:28,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:28,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:28,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:28,472 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,472 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,473 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,473 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:28,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46fb8b4f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:28,474 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:28,474 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,475 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:28,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:28,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:28,475 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,475 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,476 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:28,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:28,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:28,478 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:28,478 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:28,479 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:28,479 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:28,479 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:28,483 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,483 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,483 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,484 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,485 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:28,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,487 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,487 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,487 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,487 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,487 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,487 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,487 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:28,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:28,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:28,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:28,487 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:28,487 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:28,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360988505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360988505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360988506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360988510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,588 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:28,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:28,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:28,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:28,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:28,591 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:28,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:28,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:28,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:28,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,592 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:28,592 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:28,593 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,593 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:28,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:28,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:28,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:28,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:28,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:28,596 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ee82151[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:28,598 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:28,598 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:28,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:28,600 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:28,601 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,601 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,602 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:28,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:28,605 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:28,605 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:28,605 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:28,605 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:28,605 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:28,606 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,606 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,606 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,607 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,608 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:28,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:28,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:28,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:28,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:28,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,611 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:28,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,611 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43504 deadline: 1680360988651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,659 DEBUG [Thread-2172] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18252 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., hostname=jenkins-hbase4.apache.org,40863,1680360818567, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-01 14:55:28,711 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:28,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:28,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360988708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:28,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:28,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:28,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,714 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:28,714 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:28,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360988712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,714 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:28,714 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:28,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360988712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,717 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,717 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:28,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:28,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:28,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:28,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:28,719 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:28,719 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:28,719 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:28,719 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:28,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:28,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360988718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,722 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:28,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28232016[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:28,724 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:28,724 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,725 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:28,725 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,725 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:28,727 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:28,727 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,727 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,727 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:28,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:28,729 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:28,729 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:28,729 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:28,729 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:28,729 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:28,732 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,732 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,732 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,733 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,735 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,735 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:28,735 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,735 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,735 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,735 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:28,736 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,736 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,736 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,736 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:28,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:28,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,736 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:28,736 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:28,794 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401178a8c2ada214d7dae47b35e61fd25ed to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401178a8c2ada214d7dae47b35e61fd25ed 2023-04-01 14:55:28,836 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:28,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:28,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:28,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:28,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:28,839 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:28,839 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:28,839 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:28,839 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:28,839 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,840 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,841 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:28,841 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:28,841 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,841 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:28,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:28,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:28,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:28,842 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:28,842 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:28,842 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:28,843 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:28,843 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:28,843 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:28,843 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,845 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,845 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:28,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20209878[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:28,846 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:28,846 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,848 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:28,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:28,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:28,848 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,848 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:28,849 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:28,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:28,851 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:28,851 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:28,851 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:28,851 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:28,852 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:28,853 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,853 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,853 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,854 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,855 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:28,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,861 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:28,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:28,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:28,862 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,862 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,862 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:28,862 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:28,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,962 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:28,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:28,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:28,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:28,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:28,965 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:28,965 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:28,965 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:28,965 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:28,965 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,965 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,967 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,967 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:28,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:28,967 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,967 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:28,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:28,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:28,968 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:28,968 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:28,968 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:28,968 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:28,968 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:28,968 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:28,968 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:28,969 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:28,969 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:28,970 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,970 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,971 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,971 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:28,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a1bcec0[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:28,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:28,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,973 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,973 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:28,973 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:28,973 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:28,973 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,973 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:28,975 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:28,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:28,977 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:28,978 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:28,978 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:28,978 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:28,978 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:28,979 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,979 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,980 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,980 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,981 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:28,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:28,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:28,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:28,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:28,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:28,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:28,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:28,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:28,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:28,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:28,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:28,983 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:28,983 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:28,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:29,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:29,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360989014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:29,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360989016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:29,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360989017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:29,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360989023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,083 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:29,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:29,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:29,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:29,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:29,086 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:29,087 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,087 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:29,087 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:29,087 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,087 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,088 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:29,088 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:29,089 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,089 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:29,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:29,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:29,090 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,090 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:29,090 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:29,090 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:29,090 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,093 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:29,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4506927b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:29,094 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:29,095 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,095 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:29,096 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,096 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:29,097 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,097 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:29,097 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,097 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:29,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:29,099 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:29,099 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:29,099 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:29,099 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:29,099 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:29,100 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,100 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,101 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,101 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,102 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,106 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,106 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:29,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:29,106 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,106 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:29,106 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,106 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:29,106 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:29,106 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:29,106 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:29,106 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:29,106 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,106 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:29,199 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=406, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/165b3ef7cc844680bf7dbdbdba699c32 2023-04-01 14:55:29,206 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:29,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:29,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:29,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:29,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:29,210 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:29,210 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,210 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:29,210 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:29,210 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,210 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,211 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:29,211 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:29,212 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,212 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:29,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:29,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:29,213 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,213 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:29,213 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:29,213 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:29,213 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:29,215 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,215 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,215 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,215 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:29,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7414295d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:29,216 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:29,217 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,217 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,218 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:29,218 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:29,218 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:29,218 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,218 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:29,225 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,225 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:29,225 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,225 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:29,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:29,229 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:29,229 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:29,229 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:29,229 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:29,229 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:29,230 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,230 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,231 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:29,231 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,232 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,236 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:29,236 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,236 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,236 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:29,236 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:29,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:29,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:29,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:29,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:29,236 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:29,237 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,237 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,237 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:29,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:29,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:29,237 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:29,237 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,237 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,337 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:29,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:29,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:29,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:29,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:29,340 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:29,340 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,340 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:29,340 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:29,341 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,341 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,343 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:29,343 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:29,343 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,344 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:29,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:29,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:29,345 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,345 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:29,345 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:29,345 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:29,346 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:29,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50d39cff[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:29,348 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:29,348 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,349 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,349 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:29,349 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:29,349 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:29,349 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:29,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:29,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:29,351 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:29,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:29,353 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:29,353 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:29,353 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:29,353 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:29,353 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:29,356 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,356 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,356 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:29,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,357 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,358 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:29,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:29,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:29,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:29,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:29,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,361 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,361 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:29,361 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:29,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:29,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:29,461 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:29,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:29,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:29,465 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:29,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:29,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:29,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,466 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:29,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:29,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,467 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:29,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:29,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:29,468 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,468 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:29,468 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:29,468 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:29,469 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:29,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34da35eb[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:29,471 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:29,471 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,473 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,473 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,473 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:29,473 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:29,473 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:29,473 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:29,475 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:29,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:29,477 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:29,477 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:29,477 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:29,477 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:29,477 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:29,479 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,479 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,479 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,479 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,480 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:29,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:29,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,483 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,483 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,483 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,483 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,483 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:29,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:29,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:29,482 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:29,483 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:29,483 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:29,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:29,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:29,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:29,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43492 deadline: 1680360989520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:29,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43480 deadline: 1680360989524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:29,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43524 deadline: 1680360989526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:29,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43514 deadline: 1680360989531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,583 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:29,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:29,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:29,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:29,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:29,586 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:29,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:29,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:29,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,587 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,589 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:29,589 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:29,589 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,589 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:29,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:29,590 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:29,590 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,590 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:29,590 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:29,590 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:29,590 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,590 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:29,590 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. as already flushing 2023-04-01 14:55:29,590 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:29,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:29,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:29,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:29,592 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,592 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:29,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453eec41[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:29,594 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:29,594 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,595 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:29,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:29,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:29,595 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,595 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:29,596 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:29,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:29,598 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:29,598 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:29,598 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:29,598 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:29,598 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:29,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:29,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:29,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:29,601 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,602 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:29,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:29,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:29,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:29,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:29,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:29,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:29,604 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:29,604 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:29,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:29,604 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:29,608 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/71977a3fadbc4b8f8f001e5ed0a6ad43 2023-04-01 14:55:29,616 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/e03d1e4352ef4b4ab2c42a6546dee79d 2023-04-01 14:55:29,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/165b3ef7cc844680bf7dbdbdba699c32 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/165b3ef7cc844680bf7dbdbdba699c32 2023-04-01 14:55:29,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/165b3ef7cc844680bf7dbdbdba699c32, entries=200, sequenceid=406, filesize=32.3 K 2023-04-01 14:55:29,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/71977a3fadbc4b8f8f001e5ed0a6ad43 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/71977a3fadbc4b8f8f001e5ed0a6ad43 2023-04-01 14:55:29,626 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/71977a3fadbc4b8f8f001e5ed0a6ad43, entries=150, sequenceid=406, filesize=11.9 K 2023-04-01 14:55:29,626 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/e03d1e4352ef4b4ab2c42a6546dee79d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/e03d1e4352ef4b4ab2c42a6546dee79d 2023-04-01 14:55:29,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/e03d1e4352ef4b4ab2c42a6546dee79d, entries=150, sequenceid=406, filesize=11.9 K 2023-04-01 14:55:29,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for b2dc695f920ae42c5964c2b96bd0ec47 in 1246ms, sequenceid=406, compaction requested=true 2023-04-01 14:55:29,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:29,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:29,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:29,630 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:55:29,630 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:55:29,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:29,630 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:55:29,631 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112144 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:55:29,631 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/B is initiating minor compaction (all files) 2023-04-01 14:55:29,631 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/A is initiating minor compaction (all files) 2023-04-01 14:55:29,631 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/B in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,631 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/A in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,631 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/df26be9dfa6847c784c4d8a38f1014e3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/bc24939b25a24660be8bfcfb0a05907a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/92e1092bc87f4f01a956b70076f95a12, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/71977a3fadbc4b8f8f001e5ed0a6ad43] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=48.5 K 2023-04-01 14:55:29,631 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7410269f09f04812815b0ba2ec72b7fc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/e33e94157a8242df8ead02fa9102f4a0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/016e33eff6474a369e29ed23911f27aa, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/165b3ef7cc844680bf7dbdbdba699c32] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=109.5 K 2023-04-01 14:55:29,631 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting df26be9dfa6847c784c4d8a38f1014e3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680360924827 2023-04-01 14:55:29,631 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 7410269f09f04812815b0ba2ec72b7fc, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680360924827 2023-04-01 14:55:29,631 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting bc24939b25a24660be8bfcfb0a05907a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680360925948 2023-04-01 14:55:29,631 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting e33e94157a8242df8ead02fa9102f4a0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680360925948 2023-04-01 14:55:29,632 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 92e1092bc87f4f01a956b70076f95a12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680360927100 2023-04-01 14:55:29,632 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 016e33eff6474a369e29ed23911f27aa, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680360927100 2023-04-01 14:55:29,632 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 71977a3fadbc4b8f8f001e5ed0a6ad43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680360928272 2023-04-01 14:55:29,632 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 165b3ef7cc844680bf7dbdbdba699c32, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680360928272 2023-04-01 14:55:29,650 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#B#compaction#388 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:29,653 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#A#compaction#387 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:29,705 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:29,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:29,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:29,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:29,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:29,708 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:29,708 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:29,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:29,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,710 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:29,710 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:29,711 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:29,711 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:29,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:29,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:29,712 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:29,712 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. started... 2023-04-01 14:55:29,713 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 14:55:29,713 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:29,713 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:29,713 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:29,713 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:29,713 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:29,713 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:29,726 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040169cb2e5b653e430b8ebbe3e3b9c78b53 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040169cb2e5b653e430b8ebbe3e3b9c78b53 2023-04-01 14:55:29,736 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=419, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/537512d8b9c143259bead148eb0281c3 2023-04-01 14:55:29,748 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/5799f5faa0f44af49184caaa4b0a0bea 2023-04-01 14:55:30,048 DEBUG [Thread-2187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e079da1 to 127.0.0.1:60465 2023-04-01 14:55:30,048 DEBUG [Thread-2187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:30,048 DEBUG [Thread-2183] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:60465 2023-04-01 14:55:30,048 DEBUG [Thread-2183] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:30,049 DEBUG [Thread-2185] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:60465 2023-04-01 14:55:30,049 DEBUG [Thread-2185] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:30,053 DEBUG [Thread-2179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ccd8b7 to 127.0.0.1:60465 2023-04-01 14:55:30,053 DEBUG [Thread-2179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:30,053 DEBUG [Thread-2181] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68a0e21a to 127.0.0.1:60465 2023-04-01 14:55:30,053 DEBUG [Thread-2181] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:30,060 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/1e9019f0d9c7439596d6958207ee8be6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1e9019f0d9c7439596d6958207ee8be6 2023-04-01 14:55:30,064 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/B of b2dc695f920ae42c5964c2b96bd0ec47 into 1e9019f0d9c7439596d6958207ee8be6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:30,064 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:30,064 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/B, priority=12, startTime=1680360929630; duration=0sec 2023-04-01 14:55:30,064 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:30,064 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:55:30,065 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:55:30,065 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): b2dc695f920ae42c5964c2b96bd0ec47/C is initiating minor compaction (all files) 2023-04-01 14:55:30,065 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b2dc695f920ae42c5964c2b96bd0ec47/C in TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:30,065 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/6780f0d5d7a844a4bbf363d758a5530e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/56530be916ba489fbd22012907c560a7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/06cfd8fabd0e4acfb16b2b9ef34a4d72, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/e03d1e4352ef4b4ab2c42a6546dee79d] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp, totalSize=48.5 K 2023-04-01 14:55:30,066 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6780f0d5d7a844a4bbf363d758a5530e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680360924827 2023-04-01 14:55:30,066 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 56530be916ba489fbd22012907c560a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680360925948 2023-04-01 14:55:30,066 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 06cfd8fabd0e4acfb16b2b9ef34a4d72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680360927100 2023-04-01 14:55:30,067 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e03d1e4352ef4b4ab2c42a6546dee79d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680360928272 2023-04-01 14:55:30,071 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): b2dc695f920ae42c5964c2b96bd0ec47#C#compaction#392 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-01 14:55:30,161 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/3bfe94418e824f9fbbbed4f5c9f0111f 2023-04-01 14:55:30,165 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/537512d8b9c143259bead148eb0281c3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/537512d8b9c143259bead148eb0281c3 2023-04-01 14:55:30,168 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/537512d8b9c143259bead148eb0281c3, entries=150, sequenceid=419, filesize=25.4 K 2023-04-01 14:55:30,168 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/5799f5faa0f44af49184caaa4b0a0bea as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/5799f5faa0f44af49184caaa4b0a0bea 2023-04-01 14:55:30,172 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/5799f5faa0f44af49184caaa4b0a0bea, entries=150, sequenceid=419, filesize=11.9 K 2023-04-01 14:55:30,172 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/3bfe94418e824f9fbbbed4f5c9f0111f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/3bfe94418e824f9fbbbed4f5c9f0111f 2023-04-01 14:55:30,176 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/3bfe94418e824f9fbbbed4f5c9f0111f, entries=150, sequenceid=419, filesize=11.9 K 2023-04-01 14:55:30,176 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for b2dc695f920ae42c5964c2b96bd0ec47 in 463ms, sequenceid=419, compaction requested=false 2023-04-01 14:55:30,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:30,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:30,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:30,178 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:30,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:30,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10b57b7a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:30,179 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:30,179 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:30,180 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:30,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:30,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:30,180 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:30,181 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:30,181 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,181 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:30,182 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:30,183 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:30,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,183 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:30,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:30,184 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:30,184 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:30,184 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:30,184 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:30,184 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:30,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:30,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:30,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:30,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:30,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:30,186 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:30,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:30,187 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:30,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:30,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:30,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:30,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:30,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:30,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:30,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:30,189 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:30,189 DEBUG [Thread-2178] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:30,189 DEBUG [Thread-2178] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:30,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:30,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:30,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:30,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:30,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:30,289 DEBUG [Thread-2178] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:30,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:30,478 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/5abc8317257f41cd87c8ec1e989e580f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/5abc8317257f41cd87c8ec1e989e580f 2023-04-01 14:55:30,481 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/C of b2dc695f920ae42c5964c2b96bd0ec47 into 5abc8317257f41cd87c8ec1e989e580f(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:30,481 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:30,481 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/C, priority=12, startTime=1680360929630; duration=0sec 2023-04-01 14:55:30,482 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:30,532 DEBUG [Thread-2174] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:60465 2023-04-01 14:55:30,532 DEBUG [Thread-2174] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:30,534 DEBUG [Thread-2168] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bcfc1e0 to 127.0.0.1:60465 2023-04-01 14:55:30,534 DEBUG [Thread-2168] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:30,535 DEBUG [Thread-2176] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:60465 2023-04-01 14:55:30,535 DEBUG [Thread-2170] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:60465 2023-04-01 14:55:30,535 DEBUG [Thread-2176] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:30,535 DEBUG [Thread-2170] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:30,874 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/67fccfed78e14a2482a7ed0821c6299e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/67fccfed78e14a2482a7ed0821c6299e 2023-04-01 14:55:30,878 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b2dc695f920ae42c5964c2b96bd0ec47/A of b2dc695f920ae42c5964c2b96bd0ec47 into 67fccfed78e14a2482a7ed0821c6299e(size=26.5 K), total size for store is 51.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-01 14:55:30,878 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:30,878 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47., storeName=b2dc695f920ae42c5964c2b96bd0ec47/A, priority=12, startTime=1680360929630; duration=1sec 2023-04-01 14:55:30,878 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:38,661 DEBUG [Thread-2172] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:60465 2023-04-01 14:55:38,661 DEBUG [Thread-2172] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 125 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 66 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 8 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 62 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 2028 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 6084 rows 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 2036 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 6108 rows 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 2034 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 6102 rows 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 2033 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 6099 rows 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 2023 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 6065 rows 2023-04-01 14:55:38,662 INFO [Listener at localhost/46667] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 14:55:38,662 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a5ab074 to 127.0.0.1:60465 2023-04-01 14:55:38,662 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:55:38,664 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 14:55:38,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-01 14:55:38,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:38,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 14:55:38,670 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360938670"}]},"ts":"1680360938670"} 2023-04-01 14:55:38,671 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 14:55:38,673 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 14:55:38,673 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b2dc695f920ae42c5964c2b96bd0ec47, UNASSIGN}] 2023-04-01 14:55:38,675 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b2dc695f920ae42c5964c2b96bd0ec47, UNASSIGN 2023-04-01 14:55:38,675 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=b2dc695f920ae42c5964c2b96bd0ec47, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:38,675 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360938675"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360938675"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360938675"}]},"ts":"1680360938675"} 2023-04-01 14:55:38,676 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:55:38,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 14:55:38,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:38,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b2dc695f920ae42c5964c2b96bd0ec47, disabling compactions & flushes 2023-04-01 14:55:38,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:38,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:38,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. after waiting 0 ms 2023-04-01 14:55:38,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:38,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing b2dc695f920ae42c5964c2b96bd0ec47 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-01 14:55:38,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=A 2023-04-01 14:55:38,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:38,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=B 2023-04-01 14:55:38,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:38,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b2dc695f920ae42c5964c2b96bd0ec47, store=C 2023-04-01 14:55:38,828 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:38,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 14:55:39,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040111ef503582394c6e9fd6c799852b0473 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040111ef503582394c6e9fd6c799852b0473 2023-04-01 14:55:39,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 14:55:39,642 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=430, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/8791aae9aa1c481eba954860e564fa5f 2023-04-01 14:55:39,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 14:55:39,963 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/dbf51ab2b2984b28b3227da93b3da425, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/0abc269841ad4779a645e6a4b47a8f9c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/da9c137e305e45a8be27cfc7b52ef247, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d097dcacf5f442838ccf34feb1abd584, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7ae20c29ee57446983e6276b4d27d458, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/ef12c646d19c42e4bff25bc5666480cc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/f709dc92feb9452eb4e7384a4e24c1ac, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7239114145f44127b1d5c914dce51bdb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/1de550009af14fad838a35d224c833e8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/3b145bf88ec64bef9276cc9bc3ad10a8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/cc7425792f794ff09cf0d0a976a1f656, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/02e43b49d272474b90b86a7a9ca3dd4c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d4ca18befccc44f9b23066ec0c46753a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9312c3bfd43745de8db6530804a9c0d6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/5e1ef2fc8cec431c87bf46e37ec9a988, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6ddf3c6321d24f6db4f1b7bfb70f5db9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/44daaa7b19e4421fa8ca97910031e069, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/70824fbff9a7420894c3ac88c604847c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/43167892b82f4ec4bc9b809b99986cf0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d87242cbea7d412fa336ec9159eb4087, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/33552577916a4b91a81864f9eaeeb87d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6fb8950d2f2d4869964df842a72492c2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/98480c45c71e4a3fae9f34b42ad63bfc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/3850f0a17f214b94bb3035bc9ffb9504, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9ac50d8a297e4214ad99afbd6409fd2a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/a2ae2763003d45e6bb33c1135864e66b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/b856ddd4e12f4f4dabd40b813039a00f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7410269f09f04812815b0ba2ec72b7fc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/e33e94157a8242df8ead02fa9102f4a0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/016e33eff6474a369e29ed23911f27aa, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/165b3ef7cc844680bf7dbdbdba699c32] to archive 2023-04-01 14:55:39,963 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:55:39,965 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/dbf51ab2b2984b28b3227da93b3da425 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/dbf51ab2b2984b28b3227da93b3da425 2023-04-01 14:55:39,966 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/0abc269841ad4779a645e6a4b47a8f9c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/0abc269841ad4779a645e6a4b47a8f9c 2023-04-01 14:55:39,967 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/da9c137e305e45a8be27cfc7b52ef247 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/da9c137e305e45a8be27cfc7b52ef247 2023-04-01 14:55:39,968 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d097dcacf5f442838ccf34feb1abd584 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d097dcacf5f442838ccf34feb1abd584 2023-04-01 14:55:39,969 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7ae20c29ee57446983e6276b4d27d458 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7ae20c29ee57446983e6276b4d27d458 2023-04-01 14:55:39,969 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/ef12c646d19c42e4bff25bc5666480cc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/ef12c646d19c42e4bff25bc5666480cc 2023-04-01 14:55:39,970 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/f709dc92feb9452eb4e7384a4e24c1ac to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/f709dc92feb9452eb4e7384a4e24c1ac 2023-04-01 14:55:39,971 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7239114145f44127b1d5c914dce51bdb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7239114145f44127b1d5c914dce51bdb 2023-04-01 14:55:39,972 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/1de550009af14fad838a35d224c833e8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/1de550009af14fad838a35d224c833e8 2023-04-01 14:55:39,972 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/3b145bf88ec64bef9276cc9bc3ad10a8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/3b145bf88ec64bef9276cc9bc3ad10a8 2023-04-01 14:55:39,973 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/cc7425792f794ff09cf0d0a976a1f656 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/cc7425792f794ff09cf0d0a976a1f656 2023-04-01 14:55:39,974 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/02e43b49d272474b90b86a7a9ca3dd4c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/02e43b49d272474b90b86a7a9ca3dd4c 2023-04-01 14:55:39,974 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d4ca18befccc44f9b23066ec0c46753a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d4ca18befccc44f9b23066ec0c46753a 2023-04-01 14:55:39,975 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9312c3bfd43745de8db6530804a9c0d6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9312c3bfd43745de8db6530804a9c0d6 2023-04-01 14:55:39,976 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/5e1ef2fc8cec431c87bf46e37ec9a988 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/5e1ef2fc8cec431c87bf46e37ec9a988 2023-04-01 14:55:39,977 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6ddf3c6321d24f6db4f1b7bfb70f5db9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6ddf3c6321d24f6db4f1b7bfb70f5db9 2023-04-01 14:55:39,977 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/44daaa7b19e4421fa8ca97910031e069 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/44daaa7b19e4421fa8ca97910031e069 2023-04-01 14:55:39,978 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/70824fbff9a7420894c3ac88c604847c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/70824fbff9a7420894c3ac88c604847c 2023-04-01 14:55:39,979 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/43167892b82f4ec4bc9b809b99986cf0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/43167892b82f4ec4bc9b809b99986cf0 2023-04-01 14:55:39,979 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d87242cbea7d412fa336ec9159eb4087 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/d87242cbea7d412fa336ec9159eb4087 2023-04-01 14:55:39,980 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/33552577916a4b91a81864f9eaeeb87d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/33552577916a4b91a81864f9eaeeb87d 2023-04-01 14:55:39,980 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6fb8950d2f2d4869964df842a72492c2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/6fb8950d2f2d4869964df842a72492c2 2023-04-01 14:55:39,981 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/98480c45c71e4a3fae9f34b42ad63bfc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/98480c45c71e4a3fae9f34b42ad63bfc 2023-04-01 14:55:39,982 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/3850f0a17f214b94bb3035bc9ffb9504 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/3850f0a17f214b94bb3035bc9ffb9504 2023-04-01 14:55:39,982 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9ac50d8a297e4214ad99afbd6409fd2a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/9ac50d8a297e4214ad99afbd6409fd2a 2023-04-01 14:55:39,983 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/a2ae2763003d45e6bb33c1135864e66b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/a2ae2763003d45e6bb33c1135864e66b 2023-04-01 14:55:39,984 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/b856ddd4e12f4f4dabd40b813039a00f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/b856ddd4e12f4f4dabd40b813039a00f 2023-04-01 14:55:39,985 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7410269f09f04812815b0ba2ec72b7fc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/7410269f09f04812815b0ba2ec72b7fc 2023-04-01 14:55:39,986 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/e33e94157a8242df8ead02fa9102f4a0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/e33e94157a8242df8ead02fa9102f4a0 2023-04-01 14:55:39,986 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/016e33eff6474a369e29ed23911f27aa to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/016e33eff6474a369e29ed23911f27aa 2023-04-01 14:55:39,987 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/165b3ef7cc844680bf7dbdbdba699c32 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/165b3ef7cc844680bf7dbdbdba699c32 2023-04-01 14:55:39,993 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/efedb47336ff4e20ad763f998fb9b24b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a6b6c88c5307458bb8894230c256ffea, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/0f88d4bfe3004166bb80a20de2f78764, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a38817ccce0f40c7bd7f033cee74b3f1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2b453c9a589e4b79a4b63be64636120b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8da1459824814933956bb16ec6657af4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/9d5173f476ee4c8898810f3baaff6ea0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2f054f93f02b4a4c8f0e54571c363ee6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8e641f4519144071828d4ad7981bd924, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/3cabb2de6eaf4eda96d7022fe178f8e3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8267a639285b4177a4eafe40113e8d58, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/166931bac05540a89868ec939a57aea0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/20b52a782945484f8a7a89d35059d64a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/0eac0a24f1a946d799b8a94b10ec0701, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2dd8268f77274b09ba85f0b79cf62e94, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a1a9faa70b114fbb8de847113aa8faa2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/d0d434a13ddf479bb596cc72f31b09f7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/5615cff9c4134ea8a29fe797d4622086, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b5d8bb6fcf84474589c26a30b7f2c7ea, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/37e11e9b516942d298ebbdff25e10d40, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2ad605a4d95e4516ab68a88fe1ba01d9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/6538b64854814f7999790f8cdd2ed18d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/f073554e840747cfb1585f845183a949, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b979e7b8a51f4f488513085645483164, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/326200a0ffad4dbdbf1af5674fe7144d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1db25a37137d451a84d3a73fed5d6c44, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/df26be9dfa6847c784c4d8a38f1014e3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1296417c33e142118a3b079149bd5d96, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/bc24939b25a24660be8bfcfb0a05907a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/92e1092bc87f4f01a956b70076f95a12, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/71977a3fadbc4b8f8f001e5ed0a6ad43] to archive 2023-04-01 14:55:39,993 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:55:39,994 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/efedb47336ff4e20ad763f998fb9b24b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/efedb47336ff4e20ad763f998fb9b24b 2023-04-01 14:55:39,995 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a6b6c88c5307458bb8894230c256ffea to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a6b6c88c5307458bb8894230c256ffea 2023-04-01 14:55:39,996 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/0f88d4bfe3004166bb80a20de2f78764 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/0f88d4bfe3004166bb80a20de2f78764 2023-04-01 14:55:39,997 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a38817ccce0f40c7bd7f033cee74b3f1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a38817ccce0f40c7bd7f033cee74b3f1 2023-04-01 14:55:39,998 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2b453c9a589e4b79a4b63be64636120b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2b453c9a589e4b79a4b63be64636120b 2023-04-01 14:55:39,998 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8da1459824814933956bb16ec6657af4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8da1459824814933956bb16ec6657af4 2023-04-01 14:55:39,999 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/9d5173f476ee4c8898810f3baaff6ea0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/9d5173f476ee4c8898810f3baaff6ea0 2023-04-01 14:55:40,000 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2f054f93f02b4a4c8f0e54571c363ee6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2f054f93f02b4a4c8f0e54571c363ee6 2023-04-01 14:55:40,001 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8e641f4519144071828d4ad7981bd924 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8e641f4519144071828d4ad7981bd924 2023-04-01 14:55:40,002 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/3cabb2de6eaf4eda96d7022fe178f8e3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/3cabb2de6eaf4eda96d7022fe178f8e3 2023-04-01 14:55:40,003 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8267a639285b4177a4eafe40113e8d58 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/8267a639285b4177a4eafe40113e8d58 2023-04-01 14:55:40,003 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/166931bac05540a89868ec939a57aea0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/166931bac05540a89868ec939a57aea0 2023-04-01 14:55:40,004 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/20b52a782945484f8a7a89d35059d64a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/20b52a782945484f8a7a89d35059d64a 2023-04-01 14:55:40,005 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/0eac0a24f1a946d799b8a94b10ec0701 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/0eac0a24f1a946d799b8a94b10ec0701 2023-04-01 14:55:40,006 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2dd8268f77274b09ba85f0b79cf62e94 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2dd8268f77274b09ba85f0b79cf62e94 2023-04-01 14:55:40,006 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a1a9faa70b114fbb8de847113aa8faa2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/a1a9faa70b114fbb8de847113aa8faa2 2023-04-01 14:55:40,007 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/d0d434a13ddf479bb596cc72f31b09f7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/d0d434a13ddf479bb596cc72f31b09f7 2023-04-01 14:55:40,008 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/5615cff9c4134ea8a29fe797d4622086 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/5615cff9c4134ea8a29fe797d4622086 2023-04-01 14:55:40,009 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b5d8bb6fcf84474589c26a30b7f2c7ea to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b5d8bb6fcf84474589c26a30b7f2c7ea 2023-04-01 14:55:40,010 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/37e11e9b516942d298ebbdff25e10d40 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/37e11e9b516942d298ebbdff25e10d40 2023-04-01 14:55:40,011 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2ad605a4d95e4516ab68a88fe1ba01d9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/2ad605a4d95e4516ab68a88fe1ba01d9 2023-04-01 14:55:40,011 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/6538b64854814f7999790f8cdd2ed18d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/6538b64854814f7999790f8cdd2ed18d 2023-04-01 14:55:40,012 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/f073554e840747cfb1585f845183a949 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/f073554e840747cfb1585f845183a949 2023-04-01 14:55:40,013 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b979e7b8a51f4f488513085645483164 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/b979e7b8a51f4f488513085645483164 2023-04-01 14:55:40,014 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/326200a0ffad4dbdbf1af5674fe7144d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/326200a0ffad4dbdbf1af5674fe7144d 2023-04-01 14:55:40,014 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1db25a37137d451a84d3a73fed5d6c44 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1db25a37137d451a84d3a73fed5d6c44 2023-04-01 14:55:40,015 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/df26be9dfa6847c784c4d8a38f1014e3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/df26be9dfa6847c784c4d8a38f1014e3 2023-04-01 14:55:40,016 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1296417c33e142118a3b079149bd5d96 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1296417c33e142118a3b079149bd5d96 2023-04-01 14:55:40,016 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/bc24939b25a24660be8bfcfb0a05907a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/bc24939b25a24660be8bfcfb0a05907a 2023-04-01 14:55:40,017 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/92e1092bc87f4f01a956b70076f95a12 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/92e1092bc87f4f01a956b70076f95a12 2023-04-01 14:55:40,018 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/71977a3fadbc4b8f8f001e5ed0a6ad43 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/71977a3fadbc4b8f8f001e5ed0a6ad43 2023-04-01 14:55:40,023 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bf202651dc994e73b2ab8f84b1976654, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9175178b572b4718baa688e694332d48, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9e8e202bb31f4cb79ef21c240391fa44, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ae66db29b58145f2bef72378dec5e975, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bdf34fc4f3ce41b1a182ba7fb71bc9af, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ed660f2936be49eb877d3422eb72aeb8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/0d380cf874884bc99cac2022a97be2d6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/270f0d9c80ef40b3b630f3556f0887ab, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/5897f6631ee5466eb40f81f3af7ccdb1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ac93e911987444609a8e5acf03d11f25, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/51263beccfc3489f969108b892b055c7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/3c6294c262534eb58262d8def66d4281, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c40306e4754a4644a16b23db54ea8a52, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/483fe001c759453b948fa0a8451cbba0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c7848e140bf24b46ab755fb8933d29ba, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9b5f76bd4dc44149836f2fc753e42a1f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9e1a53cc40b54b04bd52eaa59ec914e1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9058932546d14a20978fc9308a0eb8f5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/b0d19d8ceb6942ee9ea9555b48d567f3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ac3645fe5eff423baf5e84fce14a2391, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/f5f7e81dafb8475caf645d589ec4ccc6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9373bbf3502a45b18f615694e2d444e3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/144da3f3a6f7442187144d3de92dd5b5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d00f5eecd40041c4810c36cbd3e9757f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/14d873cf3a09417f949c4755f9ff1364, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/405a31347cde4bb0974af5c429f10dfb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/6780f0d5d7a844a4bbf363d758a5530e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d1f9e24b30f8408db6d85c633f6acda8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/56530be916ba489fbd22012907c560a7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/06cfd8fabd0e4acfb16b2b9ef34a4d72, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/e03d1e4352ef4b4ab2c42a6546dee79d] to archive 2023-04-01 14:55:40,023 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:55:40,024 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bf202651dc994e73b2ab8f84b1976654 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bf202651dc994e73b2ab8f84b1976654 2023-04-01 14:55:40,025 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9175178b572b4718baa688e694332d48 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9175178b572b4718baa688e694332d48 2023-04-01 14:55:40,026 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9e8e202bb31f4cb79ef21c240391fa44 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9e8e202bb31f4cb79ef21c240391fa44 2023-04-01 14:55:40,026 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ae66db29b58145f2bef72378dec5e975 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ae66db29b58145f2bef72378dec5e975 2023-04-01 14:55:40,027 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bdf34fc4f3ce41b1a182ba7fb71bc9af to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/bdf34fc4f3ce41b1a182ba7fb71bc9af 2023-04-01 14:55:40,028 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ed660f2936be49eb877d3422eb72aeb8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ed660f2936be49eb877d3422eb72aeb8 2023-04-01 14:55:40,029 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/0d380cf874884bc99cac2022a97be2d6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/0d380cf874884bc99cac2022a97be2d6 2023-04-01 14:55:40,030 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/270f0d9c80ef40b3b630f3556f0887ab to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/270f0d9c80ef40b3b630f3556f0887ab 2023-04-01 14:55:40,030 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/5897f6631ee5466eb40f81f3af7ccdb1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/5897f6631ee5466eb40f81f3af7ccdb1 2023-04-01 14:55:40,031 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ac93e911987444609a8e5acf03d11f25 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ac93e911987444609a8e5acf03d11f25 2023-04-01 14:55:40,032 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/51263beccfc3489f969108b892b055c7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/51263beccfc3489f969108b892b055c7 2023-04-01 14:55:40,033 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/3c6294c262534eb58262d8def66d4281 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/3c6294c262534eb58262d8def66d4281 2023-04-01 14:55:40,033 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c40306e4754a4644a16b23db54ea8a52 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c40306e4754a4644a16b23db54ea8a52 2023-04-01 14:55:40,034 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/483fe001c759453b948fa0a8451cbba0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/483fe001c759453b948fa0a8451cbba0 2023-04-01 14:55:40,035 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c7848e140bf24b46ab755fb8933d29ba to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/c7848e140bf24b46ab755fb8933d29ba 2023-04-01 14:55:40,036 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9b5f76bd4dc44149836f2fc753e42a1f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9b5f76bd4dc44149836f2fc753e42a1f 2023-04-01 14:55:40,037 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9e1a53cc40b54b04bd52eaa59ec914e1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9e1a53cc40b54b04bd52eaa59ec914e1 2023-04-01 14:55:40,037 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9058932546d14a20978fc9308a0eb8f5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9058932546d14a20978fc9308a0eb8f5 2023-04-01 14:55:40,038 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/b0d19d8ceb6942ee9ea9555b48d567f3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/b0d19d8ceb6942ee9ea9555b48d567f3 2023-04-01 14:55:40,039 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ac3645fe5eff423baf5e84fce14a2391 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/ac3645fe5eff423baf5e84fce14a2391 2023-04-01 14:55:40,040 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/f5f7e81dafb8475caf645d589ec4ccc6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/f5f7e81dafb8475caf645d589ec4ccc6 2023-04-01 14:55:40,040 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9373bbf3502a45b18f615694e2d444e3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/9373bbf3502a45b18f615694e2d444e3 2023-04-01 14:55:40,041 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/144da3f3a6f7442187144d3de92dd5b5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/144da3f3a6f7442187144d3de92dd5b5 2023-04-01 14:55:40,042 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d00f5eecd40041c4810c36cbd3e9757f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d00f5eecd40041c4810c36cbd3e9757f 2023-04-01 14:55:40,043 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/14d873cf3a09417f949c4755f9ff1364 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/14d873cf3a09417f949c4755f9ff1364 2023-04-01 14:55:40,043 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/405a31347cde4bb0974af5c429f10dfb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/405a31347cde4bb0974af5c429f10dfb 2023-04-01 14:55:40,044 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/6780f0d5d7a844a4bbf363d758a5530e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/6780f0d5d7a844a4bbf363d758a5530e 2023-04-01 14:55:40,045 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d1f9e24b30f8408db6d85c633f6acda8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/d1f9e24b30f8408db6d85c633f6acda8 2023-04-01 14:55:40,045 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/56530be916ba489fbd22012907c560a7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/56530be916ba489fbd22012907c560a7 2023-04-01 14:55:40,046 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/06cfd8fabd0e4acfb16b2b9ef34a4d72 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/06cfd8fabd0e4acfb16b2b9ef34a4d72 2023-04-01 14:55:40,047 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/e03d1e4352ef4b4ab2c42a6546dee79d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/e03d1e4352ef4b4ab2c42a6546dee79d 2023-04-01 14:55:40,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/87429fb3773141ec9b2645ace82c6f63 2023-04-01 14:55:40,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/7cb6704b4c6c42708fd6111a50a6ee7b 2023-04-01 14:55:40,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/A/8791aae9aa1c481eba954860e564fa5f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/8791aae9aa1c481eba954860e564fa5f 2023-04-01 14:55:40,467 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/8791aae9aa1c481eba954860e564fa5f, entries=150, sequenceid=430, filesize=25.4 K 2023-04-01 14:55:40,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/B/87429fb3773141ec9b2645ace82c6f63 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/87429fb3773141ec9b2645ace82c6f63 2023-04-01 14:55:40,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/87429fb3773141ec9b2645ace82c6f63, entries=150, sequenceid=430, filesize=11.9 K 2023-04-01 14:55:40,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/.tmp/C/7cb6704b4c6c42708fd6111a50a6ee7b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/7cb6704b4c6c42708fd6111a50a6ee7b 2023-04-01 14:55:40,472 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/7cb6704b4c6c42708fd6111a50a6ee7b, entries=150, sequenceid=430, filesize=11.9 K 2023-04-01 14:55:40,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for b2dc695f920ae42c5964c2b96bd0ec47 in 1645ms, sequenceid=430, compaction requested=true 2023-04-01 14:55:40,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/recovered.edits/433.seqid, newMaxSeqId=433, maxSeqId=4 2023-04-01 14:55:40,480 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47. 2023-04-01 14:55:40,480 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b2dc695f920ae42c5964c2b96bd0ec47: 2023-04-01 14:55:40,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:40,482 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=b2dc695f920ae42c5964c2b96bd0ec47, regionState=CLOSED 2023-04-01 14:55:40,482 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360940482"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360940482"}]},"ts":"1680360940482"} 2023-04-01 14:55:40,484 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-01 14:55:40,484 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure b2dc695f920ae42c5964c2b96bd0ec47, server=jenkins-hbase4.apache.org,40863,1680360818567 in 1.8070 sec 2023-04-01 14:55:40,485 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-01 14:55:40,485 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b2dc695f920ae42c5964c2b96bd0ec47, UNASSIGN in 1.8110 sec 2023-04-01 14:55:40,486 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360940486"}]},"ts":"1680360940486"} 2023-04-01 14:55:40,486 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 14:55:40,488 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 14:55:40,489 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8230 sec 2023-04-01 14:55:40,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-01 14:55:40,777 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-01 14:55:40,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-01 14:55:40,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:40,779 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:40,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-01 14:55:40,780 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:40,783 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:40,784 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/recovered.edits] 2023-04-01 14:55:40,786 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/537512d8b9c143259bead148eb0281c3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/537512d8b9c143259bead148eb0281c3 2023-04-01 14:55:40,787 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/67fccfed78e14a2482a7ed0821c6299e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/67fccfed78e14a2482a7ed0821c6299e 2023-04-01 14:55:40,788 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/8791aae9aa1c481eba954860e564fa5f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/A/8791aae9aa1c481eba954860e564fa5f 2023-04-01 14:55:40,790 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1e9019f0d9c7439596d6958207ee8be6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/1e9019f0d9c7439596d6958207ee8be6 2023-04-01 14:55:40,791 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/5799f5faa0f44af49184caaa4b0a0bea to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/5799f5faa0f44af49184caaa4b0a0bea 2023-04-01 14:55:40,791 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/87429fb3773141ec9b2645ace82c6f63 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/B/87429fb3773141ec9b2645ace82c6f63 2023-04-01 14:55:40,793 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/3bfe94418e824f9fbbbed4f5c9f0111f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/3bfe94418e824f9fbbbed4f5c9f0111f 2023-04-01 14:55:40,794 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/5abc8317257f41cd87c8ec1e989e580f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/5abc8317257f41cd87c8ec1e989e580f 2023-04-01 14:55:40,795 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/7cb6704b4c6c42708fd6111a50a6ee7b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/C/7cb6704b4c6c42708fd6111a50a6ee7b 2023-04-01 14:55:40,796 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/recovered.edits/433.seqid to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47/recovered.edits/433.seqid 2023-04-01 14:55:40,797 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/b2dc695f920ae42c5964c2b96bd0ec47 2023-04-01 14:55:40,797 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:55:40,797 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 14:55:40,798 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-01 14:55:40,800 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304010d2fd944fe784e2394294cd3e899b8da to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304010d2fd944fe784e2394294cd3e899b8da 2023-04-01 14:55:40,800 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040111ef503582394c6e9fd6c799852b0473 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040111ef503582394c6e9fd6c799852b0473 2023-04-01 14:55:40,801 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011344f6c6db324a5f8e7bd229516ec2fc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011344f6c6db324a5f8e7bd229516ec2fc 2023-04-01 14:55:40,802 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401149596dae2db454a832d421fef7f9ed1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401149596dae2db454a832d421fef7f9ed1 2023-04-01 14:55:40,803 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401178a8c2ada214d7dae47b35e61fd25ed to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401178a8c2ada214d7dae47b35e61fd25ed 2023-04-01 14:55:40,803 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012eb85744af1b491d93488186acf31830 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012eb85744af1b491d93488186acf31830 2023-04-01 14:55:40,804 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040136f361823d3c4d33ad52b7d311ea53a2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040136f361823d3c4d33ad52b7d311ea53a2 2023-04-01 14:55:40,805 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013fdd44d537634d7aa6e1ab809659fda4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013fdd44d537634d7aa6e1ab809659fda4 2023-04-01 14:55:40,805 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304014c1453852413498580001bf5f3f04722 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304014c1453852413498580001bf5f3f04722 2023-04-01 14:55:40,806 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040169cb2e5b653e430b8ebbe3e3b9c78b53 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040169cb2e5b653e430b8ebbe3e3b9c78b53 2023-04-01 14:55:40,807 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040188c27cd356414b67b1b57ee9545b5421 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040188c27cd356414b67b1b57ee9545b5421 2023-04-01 14:55:40,808 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019e3f331af7be4ddc85301ab93afc1637 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019e3f331af7be4ddc85301ab93afc1637 2023-04-01 14:55:40,808 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019e49d26506d94d26abd538ad5e9d9fe0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304019e49d26506d94d26abd538ad5e9d9fe0 2023-04-01 14:55:40,809 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a93eb5ea24054b7fa7054c904ea86b65 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401a93eb5ea24054b7fa7054c904ea86b65 2023-04-01 14:55:40,810 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ba8b568ac46541e2b1593c5f6f787fcc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ba8b568ac46541e2b1593c5f6f787fcc 2023-04-01 14:55:40,811 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401bb7a20ae0f54410cab3304dbbbdd0dc6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401bb7a20ae0f54410cab3304dbbbdd0dc6 2023-04-01 14:55:40,811 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401bbfc826baf4f49b18fcd0eb630a99669 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401bbfc826baf4f49b18fcd0eb630a99669 2023-04-01 14:55:40,812 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c11ceac00a9046fe99cf96772146dea2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c11ceac00a9046fe99cf96772146dea2 2023-04-01 14:55:40,813 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c20402fd6bff4e0eac3ebfa064fbaaa6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c20402fd6bff4e0eac3ebfa064fbaaa6 2023-04-01 14:55:40,814 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401cdd08173719f499b9c8c4d73ac2a2fc9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401cdd08173719f499b9c8c4d73ac2a2fc9 2023-04-01 14:55:40,815 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401da40e695c24248df98cb25de9b3bf05a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401da40e695c24248df98cb25de9b3bf05a 2023-04-01 14:55:40,816 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f413cef780ba40ff8c7937b3586b61c0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f413cef780ba40ff8c7937b3586b61c0 2023-04-01 14:55:40,816 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f50c6f18a9734e1d8b41ab7f4f89f909 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401f50c6f18a9734e1d8b41ab7f4f89f909 2023-04-01 14:55:40,817 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401fe8bd207ec0b4bcf8fa0975a7faa667a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401fe8bd207ec0b4bcf8fa0975a7faa667a 2023-04-01 14:55:40,818 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ff0872a37bf34fbd80e1fb48706d2a6a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ff0872a37bf34fbd80e1fb48706d2a6a 2023-04-01 14:55:40,818 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 14:55:40,820 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:40,821 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 14:55:40,822 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 14:55:40,823 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:40,823 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 14:55:40,823 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680360940823"}]},"ts":"9223372036854775807"} 2023-04-01 14:55:40,824 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 14:55:40,824 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b2dc695f920ae42c5964c2b96bd0ec47, NAME => 'TestAcidGuarantees,,1680360906976.b2dc695f920ae42c5964c2b96bd0ec47.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 14:55:40,825 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 14:55:40,825 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680360940825"}]},"ts":"9223372036854775807"} 2023-04-01 14:55:40,826 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 14:55:40,827 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:40,828 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 50 msec 2023-04-01 14:55:40,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-01 14:55:40,881 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-01 14:55:40,892 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229 (was 224) - Thread LEAK? -, OpenFileDescriptor=524 (was 521) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=360 (was 363), ProcessCount=172 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=3140 (was 3278) 2023-04-01 14:55:40,903 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229, OpenFileDescriptor=524, MaxFileDescriptor=60000, SystemLoadAverage=360, ProcessCount=172, AvailableMemoryMB=3139 2023-04-01 14:55:40,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 14:55:40,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 14:55:40,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 14:55:40,907 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 14:55:40,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-01 14:55:40,908 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 14:55:40,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-01 14:55:40,908 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:40,909 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15 empty. 2023-04-01 14:55:40,909 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:40,909 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:55:41,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-01 14:55:41,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-01 14:55:41,316 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 14:55:41,317 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2ddd6fcff99c4aa857940e9dc8e40f15, NAME => 'TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp 2023-04-01 14:55:41,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-01 14:55:41,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:55:41,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2ddd6fcff99c4aa857940e9dc8e40f15, disabling compactions & flushes 2023-04-01 14:55:41,723 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:41,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:41,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. after waiting 0 ms 2023-04-01 14:55:41,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:41,723 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:41,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:41,725 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 14:55:41,726 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360941726"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360941726"}]},"ts":"1680360941726"} 2023-04-01 14:55:41,727 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 14:55:41,727 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 14:55:41,727 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360941727"}]},"ts":"1680360941727"} 2023-04-01 14:55:41,728 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 14:55:41,732 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ddd6fcff99c4aa857940e9dc8e40f15, ASSIGN}] 2023-04-01 14:55:41,733 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ddd6fcff99c4aa857940e9dc8e40f15, ASSIGN 2023-04-01 14:55:41,734 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ddd6fcff99c4aa857940e9dc8e40f15, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40863,1680360818567; forceNewPlan=false, retain=false 2023-04-01 14:55:41,885 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=2ddd6fcff99c4aa857940e9dc8e40f15, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:41,885 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360941885"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360941885"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360941885"}]},"ts":"1680360941885"} 2023-04-01 14:55:41,886 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:55:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-01 14:55:42,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:42,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2ddd6fcff99c4aa857940e9dc8e40f15, NAME => 'TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:55:42,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:42,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:55:42,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:42,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:42,041 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:42,042 DEBUG [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A 2023-04-01 14:55:42,042 DEBUG [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A 2023-04-01 14:55:42,042 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:55:42,043 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ddd6fcff99c4aa857940e9dc8e40f15 columnFamilyName A 2023-04-01 14:55:42,043 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] regionserver.HStore(310): Store=2ddd6fcff99c4aa857940e9dc8e40f15/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:55:42,043 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:42,044 DEBUG [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B 2023-04-01 14:55:42,044 DEBUG [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B 2023-04-01 14:55:42,044 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:55:42,044 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ddd6fcff99c4aa857940e9dc8e40f15 columnFamilyName B 2023-04-01 14:55:42,045 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] regionserver.HStore(310): Store=2ddd6fcff99c4aa857940e9dc8e40f15/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:55:42,045 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:42,046 DEBUG [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C 2023-04-01 14:55:42,046 DEBUG [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C 2023-04-01 14:55:42,046 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:55:42,046 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ddd6fcff99c4aa857940e9dc8e40f15 columnFamilyName C 2023-04-01 14:55:42,047 INFO [StoreOpener-2ddd6fcff99c4aa857940e9dc8e40f15-1] regionserver.HStore(310): Store=2ddd6fcff99c4aa857940e9dc8e40f15/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:55:42,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:42,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:42,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:42,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 14:55:42,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:42,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 14:55:42,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2ddd6fcff99c4aa857940e9dc8e40f15; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59239979, jitterRate=-0.11725552380084991}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 14:55:42,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:42,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., pid=49, masterSystemTime=1680360942037 2023-04-01 14:55:42,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:42,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:42,054 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=2ddd6fcff99c4aa857940e9dc8e40f15, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:42,054 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360942054"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680360942054"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680360942054"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680360942054"}]},"ts":"1680360942054"} 2023-04-01 14:55:42,055 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-01 14:55:42,055 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 in 168 msec 2023-04-01 14:55:42,056 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-01 14:55:42,056 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ddd6fcff99c4aa857940e9dc8e40f15, ASSIGN in 323 msec 2023-04-01 14:55:42,057 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 14:55:42,057 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360942057"}]},"ts":"1680360942057"} 2023-04-01 14:55:42,057 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 14:55:42,059 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 14:55:42,060 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1540 sec 2023-04-01 14:55:43,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-01 14:55:43,012 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-01 14:55:43,013 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec943a9 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:43,018 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27f2d997, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:43,019 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:43,021 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:43,022 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 14:55:43,023 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 14:55:43,025 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x20ed9117 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:43,028 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ad4aa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:43,028 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x50002343 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:43,033 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c29757d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:43,034 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:43,037 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:43,037 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:43,040 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:43,041 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:43,044 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:43,044 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x63093e21 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:43,049 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f5363d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:43,049 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:43,052 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:43,053 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:43,059 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:43,059 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed183a4 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:43,063 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@653bb8fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:43,063 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x10a344e6 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:55:43,070 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@720d29f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:55:43,078 DEBUG [hconnection-0x3197019a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:43,080 DEBUG [hconnection-0x2abd65e3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:43,080 DEBUG [hconnection-0x2bf3bc68-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:43,081 DEBUG [hconnection-0x671b6391-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:43,081 DEBUG [hconnection-0x279d72ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:43,081 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48032, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:43,082 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:43,082 DEBUG [hconnection-0xfbbc2fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:43,085 DEBUG [hconnection-0x4f6d84b8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:43,085 DEBUG [hconnection-0x43cc87cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:43,087 DEBUG [hconnection-0x2be51aad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:43,088 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48086, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:43,087 DEBUG [hconnection-0x697c440e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:55:43,089 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:43,090 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:43,089 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48024, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:43,092 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48100, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:43,092 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48048, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:43,093 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:43,098 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48116, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:55:43,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:43,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:43,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:43,103 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:43,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:43,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:43,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:43,105 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:55:43,106 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,106 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,107 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,107 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,107 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,107 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:43,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,107 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:43,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:43,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:43,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:43,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:43,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:43,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:43,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:43,108 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:43,108 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,108 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:43,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:43,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:43,109 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,109 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:43,109 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:43,109 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:43,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:43,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:43,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:43,111 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,111 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,111 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,111 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:43,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4d39f7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:43,114 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:43,115 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,116 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,116 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:43,116 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:43,116 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:43,116 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,116 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:43,118 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,119 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:43,119 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,119 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:43,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:43,121 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:43,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:43,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:43,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:43,121 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:43,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,123 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,125 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,127 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:43,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:43,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,127 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:43,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,127 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:43,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361003123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361003123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361003124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361003125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361003126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,227 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:43,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:43,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:43,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361003230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361003230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361003231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361003231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361003231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:43,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:43,234 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:43,234 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,234 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:43,234 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:43,235 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,235 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,240 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,241 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,241 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:43,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:43,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:43,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:43,242 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:43,242 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,242 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:43,242 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:43,242 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:43,242 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,243 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:43,243 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:43,243 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,243 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:43,243 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:43,243 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,243 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:43,243 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:43,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,244 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:43,247 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:43,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39dd2af8[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:43,247 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,248 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,248 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:43,248 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:43,248 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:43,248 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,248 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:43,249 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,249 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:43,249 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:43,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:43,252 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:43,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:43,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:43,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:43,252 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:43,254 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,254 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,254 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,255 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,256 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:43,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:43,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:43,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,258 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,259 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,259 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,259 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:43,259 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:43,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,359 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:43,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:43,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:43,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:43,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:43,363 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:43,363 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,363 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:43,363 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:43,364 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,364 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,365 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,365 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,365 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:43,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:43,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:43,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:43,367 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,367 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:43,367 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:43,367 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:43,367 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,368 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:43,368 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,369 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:43,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b338566[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:43,371 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:43,371 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,373 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,373 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:43,373 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:43,373 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:43,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:43,375 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:43,377 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:43,377 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:43,377 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:43,377 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:43,377 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:43,378 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,378 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,379 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,380 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,382 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,382 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,382 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:43,382 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:43,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:43,382 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,382 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,383 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:43,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,383 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:43,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361003433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361003433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361003433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361003433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361003433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,483 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:43,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:43,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:43,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:43,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:43,487 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:43,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:43,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:43,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,490 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,490 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,490 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,490 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:43,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,491 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:43,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:43,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:43,492 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,492 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:43,492 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:43,492 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:43,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:43,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:43,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:43,494 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:43,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@306641d1[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:43,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:43,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,497 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,497 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:43,497 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:43,497 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:43,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:43,498 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,498 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,498 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,498 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:43,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:43,501 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:43,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:43,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:43,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:43,501 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:43,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,504 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,505 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,507 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:43,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:43,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,508 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:43,508 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:43,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:43,518 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/3f280e4375b74a8f925245d8cf4b4116 2023-04-01 14:55:43,530 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/5164830874fd4eb5b09723a6f572640c 2023-04-01 14:55:43,608 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:43,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:43,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:43,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:43,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:43,612 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:43,612 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,612 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:43,612 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:43,613 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,613 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,614 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,614 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,614 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,614 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:43,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:43,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:43,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:43,616 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:43,616 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:43,616 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,616 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:43,616 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:43,616 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,616 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:43,616 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:43,616 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,616 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:43,616 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:43,621 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:43,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d144ab[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:43,626 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:43,626 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,627 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:43,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:43,627 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:43,629 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,629 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:43,630 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:43,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:43,632 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:43,633 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:43,633 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:43,633 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:43,633 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:43,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,635 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,637 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:43,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:43,641 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,642 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:43,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:43,642 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,642 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:43,642 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361003735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361003736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361003738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361003738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:43,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361003739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,742 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:43,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:43,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:43,746 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:43,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:43,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:43,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,748 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,748 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,749 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,749 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:43,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:43,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:43,750 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:43,750 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,750 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:43,751 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:43,751 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:43,751 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,751 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:43,751 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:43,751 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:43,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:43,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@550b2403[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:43,755 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:43,755 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,756 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:43,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:43,756 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,756 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:43,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:43,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:43,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:43,761 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:43,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:43,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:43,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:43,761 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:43,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,763 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,764 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,766 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,766 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,767 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:43,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:43,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:43,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,767 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:43,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,767 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:43,867 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:43,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:43,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:43,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:43,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:43,871 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:43,871 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,871 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:43,871 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:43,871 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,871 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,872 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,872 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,873 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,873 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:43,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:43,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:43,874 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,874 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:43,874 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:43,874 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:43,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:43,877 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:43,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42848371[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:43,879 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:43,879 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,880 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:43,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:43,881 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:43,881 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,881 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,882 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:43,884 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:43,884 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:43,884 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:43,884 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:43,884 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:43,886 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,886 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,886 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:43,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:43,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:43,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:43,891 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,891 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,891 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,891 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,891 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:43,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:43,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:43,891 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:43,891 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,891 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,891 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:43,891 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,891 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:43,891 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:43,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:43,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:43,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/143d884e4a3e419ab77f90b5e14010ff 2023-04-01 14:55:43,947 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/3f280e4375b74a8f925245d8cf4b4116 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3f280e4375b74a8f925245d8cf4b4116 2023-04-01 14:55:43,949 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3f280e4375b74a8f925245d8cf4b4116, entries=150, sequenceid=13, filesize=11.6 K 2023-04-01 14:55:43,950 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/5164830874fd4eb5b09723a6f572640c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5164830874fd4eb5b09723a6f572640c 2023-04-01 14:55:43,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5164830874fd4eb5b09723a6f572640c, entries=150, sequenceid=13, filesize=11.6 K 2023-04-01 14:55:43,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/143d884e4a3e419ab77f90b5e14010ff as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/143d884e4a3e419ab77f90b5e14010ff 2023-04-01 14:55:43,956 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/143d884e4a3e419ab77f90b5e14010ff, entries=150, sequenceid=13, filesize=11.6 K 2023-04-01 14:55:43,957 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-01 14:55:43,957 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 852ms, sequenceid=13, compaction requested=false 2023-04-01 14:55:43,957 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:43,991 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:43,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:43,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:43,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:43,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:43,995 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:43,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:43,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:43,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,996 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:43,996 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,997 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:43,997 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:43,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:43,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:43,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:43,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:43,998 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:43,998 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:43,998 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:43,999 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:43,999 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:43,999 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:43,999 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:44,000 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 14:55:44,000 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:44,000 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:44,000 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:44,000 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:44,000 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:44,000 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:44,012 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/aa9352d72262437198984120feb40433 2023-04-01 14:55:44,023 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/2767b0a29fce44439c4482d78cf98441 2023-04-01 14:55:44,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:44,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:44,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361004249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361004250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361004273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361004273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361004273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361004374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361004374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361004379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361004379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361004379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,437 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/28542dd054224b80b465225eb60f205c 2023-04-01 14:55:44,440 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/aa9352d72262437198984120feb40433 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aa9352d72262437198984120feb40433 2023-04-01 14:55:44,444 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aa9352d72262437198984120feb40433, entries=150, sequenceid=33, filesize=11.6 K 2023-04-01 14:55:44,445 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/2767b0a29fce44439c4482d78cf98441 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2767b0a29fce44439c4482d78cf98441 2023-04-01 14:55:44,447 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2767b0a29fce44439c4482d78cf98441, entries=150, sequenceid=33, filesize=11.6 K 2023-04-01 14:55:44,448 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/28542dd054224b80b465225eb60f205c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/28542dd054224b80b465225eb60f205c 2023-04-01 14:55:44,451 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/28542dd054224b80b465225eb60f205c, entries=150, sequenceid=33, filesize=11.6 K 2023-04-01 14:55:44,452 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 453ms, sequenceid=33, compaction requested=false 2023-04-01 14:55:44,452 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:44,452 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:44,452 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:44,452 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:44,452 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,452 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:44,452 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:44,453 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,453 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:44,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40ee3a39[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:44,455 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:44,455 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:44,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:44,457 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:44,457 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,457 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:44,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,466 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:44,467 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:44,467 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:44,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:44,471 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:44,471 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:44,471 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:44,471 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:44,471 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:44,472 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,472 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,472 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:44,473 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,475 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,478 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,478 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,478 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,478 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,478 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:44,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:44,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:44,478 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,478 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,478 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:44,478 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:44,478 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:44,478 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:44,478 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:44,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:44,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:44,578 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:44,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:44,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:44,580 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:44,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:44,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:44,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:44,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:44,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:44,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:44,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:44,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:44,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:44,582 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:44,582 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:44,582 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:44,583 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:44,583 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,583 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,585 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,585 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,585 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:44,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:44,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:44,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:44,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:44,587 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:44,587 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:44,587 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:44,587 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:44,587 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:44,587 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:44,587 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:44,588 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:44,588 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:44,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:44,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:44,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:44,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:44,589 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/be361e4697ee498f9a3614abb8773560 2023-04-01 14:55:44,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:44,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:44,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f95ca93[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:44,593 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:44,593 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,594 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:44,595 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,595 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:44,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:44,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:44,596 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:44,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:44,599 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:44,599 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:44,599 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:44,599 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:44,599 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:44,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:44,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:44,601 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,602 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,605 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,606 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,606 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,606 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:44,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:44,606 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,606 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:44,606 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,606 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,606 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:44,606 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:44,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:44,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:44,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:44,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:44,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:44,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:44,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361004610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361004611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361004611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361004611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361004612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,706 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:44,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:44,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:44,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:44,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:44,709 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:44,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:44,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:44,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:44,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,710 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,710 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:44,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:44,711 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,711 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:44,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:44,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:44,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:44,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:44,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:44,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:44,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:44,712 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:44,712 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:44,712 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:44,712 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:44,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:44,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:44,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:44,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:44,714 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,714 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,714 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,714 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:44,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43b2d45e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:44,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:44,716 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,716 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,717 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:44,717 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:44,717 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:44,717 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,717 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:44,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:44,720 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:44,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361004717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,722 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:44,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:44,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:44,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:44,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:44,722 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:44,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361004717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361004718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,724 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,724 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361004721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361004722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,724 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:44,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:44,725 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,725 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:44,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:44,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:44,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:44,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:44,729 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:44,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:44,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:44,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:44,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:44,729 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:44,729 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:44,829 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:44,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:44,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:44,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:44,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:44,832 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:44,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:44,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:44,833 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:44,833 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,833 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,834 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:44,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:44,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:44,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:44,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:44,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:44,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:44,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:44,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:44,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:44,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:44,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:44,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,839 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25cb4852[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:44,841 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:44,841 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,842 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,842 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:44,842 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:44,842 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,842 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:44,842 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:44,843 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:44,843 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:44,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:44,846 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:44,846 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:44,846 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:44,846 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:44,846 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:44,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:44,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:44,856 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,858 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,860 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:44,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:44,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:44,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,860 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,860 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:44,860 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:44,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:44,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:44,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:44,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:44,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:44,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361004924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361004928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361004928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361004928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:44,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361004928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,960 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:44,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:44,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:44,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:44,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:44,963 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:44,964 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:44,964 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:44,964 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:44,964 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,964 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:44,965 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:44,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:44,966 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,966 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:44,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:44,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:44,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:44,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:44,967 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:44,967 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:44,967 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:44,967 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:44,972 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:44,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14459120[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:44,975 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:44,976 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,978 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,978 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:44,981 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,981 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:44,981 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,981 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:44,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:44,984 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:44,985 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:44,985 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:44,985 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:44,985 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:44,986 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,986 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,986 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:44,986 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,988 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:44,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:44,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:44,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:44,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:44,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:44,990 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:44,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:44,990 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:44,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:44,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:44,991 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:44,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:44,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:44,998 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/9a50edac565f43a69dce8314b1ba9d02 2023-04-01 14:55:45,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/4e8f253466aa444eba76dd3739794a49 2023-04-01 14:55:45,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/be361e4697ee498f9a3614abb8773560 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/be361e4697ee498f9a3614abb8773560 2023-04-01 14:55:45,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/be361e4697ee498f9a3614abb8773560, entries=250, sequenceid=48, filesize=16.2 K 2023-04-01 14:55:45,013 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/9a50edac565f43a69dce8314b1ba9d02 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/9a50edac565f43a69dce8314b1ba9d02 2023-04-01 14:55:45,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/9a50edac565f43a69dce8314b1ba9d02, entries=150, sequenceid=48, filesize=11.6 K 2023-04-01 14:55:45,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/4e8f253466aa444eba76dd3739794a49 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4e8f253466aa444eba76dd3739794a49 2023-04-01 14:55:45,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4e8f253466aa444eba76dd3739794a49, entries=150, sequenceid=48, filesize=11.6 K 2023-04-01 14:55:45,020 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=107.34 KB/109920 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 440ms, sequenceid=48, compaction requested=true 2023-04-01 14:55:45,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:45,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:45,021 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:45,021 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:45,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:45,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:45,021 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:45,022 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/A is initiating minor compaction (all files) 2023-04-01 14:55:45,022 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:45,022 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/A in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:45,022 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:55:45,022 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3f280e4375b74a8f925245d8cf4b4116, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aa9352d72262437198984120feb40433, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/be361e4697ee498f9a3614abb8773560] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=39.4 K 2023-04-01 14:55:45,022 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:45,022 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5164830874fd4eb5b09723a6f572640c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2767b0a29fce44439c4482d78cf98441, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/9a50edac565f43a69dce8314b1ba9d02] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=34.8 K 2023-04-01 14:55:45,022 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 3f280e4375b74a8f925245d8cf4b4116, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680360943103 2023-04-01 14:55:45,022 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 5164830874fd4eb5b09723a6f572640c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680360943103 2023-04-01 14:55:45,022 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting aa9352d72262437198984120feb40433, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680360943121 2023-04-01 14:55:45,022 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2767b0a29fce44439c4482d78cf98441, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680360943121 2023-04-01 14:55:45,023 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting be361e4697ee498f9a3614abb8773560, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680360944249 2023-04-01 14:55:45,023 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9a50edac565f43a69dce8314b1ba9d02, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680360944249 2023-04-01 14:55:45,028 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#A#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:45,029 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:45,091 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:45,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:45,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:45,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:45,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:45,094 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:45,094 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:45,094 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:45,094 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:45,094 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,094 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,096 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:45,096 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:45,097 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,097 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:45,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:45,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:45,098 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:45,098 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:45,098 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:45,098 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:45,098 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:45,098 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:45,098 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:45,098 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:55:45,098 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:45,099 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:45,099 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:45,099 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:45,099 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:45,099 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:45,111 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/7e577242145e42578af6b53d80aa7d31 2023-04-01 14:55:45,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:45,231 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:45,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361005244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361005245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361005245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361005250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361005250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361005354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361005354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361005354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361005354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361005354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,440 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/cd74b407566949008d703f7eec4995e8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/cd74b407566949008d703f7eec4995e8 2023-04-01 14:55:45,446 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/A of 2ddd6fcff99c4aa857940e9dc8e40f15 into cd74b407566949008d703f7eec4995e8(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:45,446 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:45,446 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/A, priority=13, startTime=1680360945020; duration=0sec 2023-04-01 14:55:45,446 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:45,446 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:45,447 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/ef2d03f682b548719f35e193f5f8d8cb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ef2d03f682b548719f35e193f5f8d8cb 2023-04-01 14:55:45,448 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:45,448 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/C is initiating minor compaction (all files) 2023-04-01 14:55:45,448 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/C in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:45,448 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/143d884e4a3e419ab77f90b5e14010ff, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/28542dd054224b80b465225eb60f205c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4e8f253466aa444eba76dd3739794a49] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=34.8 K 2023-04-01 14:55:45,448 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 143d884e4a3e419ab77f90b5e14010ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680360943103 2023-04-01 14:55:45,449 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 28542dd054224b80b465225eb60f205c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680360943121 2023-04-01 14:55:45,449 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 4e8f253466aa444eba76dd3739794a49, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680360944249 2023-04-01 14:55:45,451 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into ef2d03f682b548719f35e193f5f8d8cb(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:45,451 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:45,451 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=13, startTime=1680360945021; duration=0sec 2023-04-01 14:55:45,451 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:45,456 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#C#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:45,470 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/6a8131293d624cf0b17a39fc81c8cdce as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/6a8131293d624cf0b17a39fc81c8cdce 2023-04-01 14:55:45,474 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/C of 2ddd6fcff99c4aa857940e9dc8e40f15 into 6a8131293d624cf0b17a39fc81c8cdce(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:45,474 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:45,474 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/C, priority=13, startTime=1680360945021; duration=0sec 2023-04-01 14:55:45,474 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:45,521 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/d4256e958bb644d091fc649b540b3719 2023-04-01 14:55:45,533 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/45b45f0fe50547d29a84f0a9de263fde 2023-04-01 14:55:45,537 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/7e577242145e42578af6b53d80aa7d31 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7e577242145e42578af6b53d80aa7d31 2023-04-01 14:55:45,540 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7e577242145e42578af6b53d80aa7d31, entries=150, sequenceid=67, filesize=11.6 K 2023-04-01 14:55:45,541 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/d4256e958bb644d091fc649b540b3719 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/d4256e958bb644d091fc649b540b3719 2023-04-01 14:55:45,544 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/d4256e958bb644d091fc649b540b3719, entries=150, sequenceid=67, filesize=11.6 K 2023-04-01 14:55:45,544 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/45b45f0fe50547d29a84f0a9de263fde as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/45b45f0fe50547d29a84f0a9de263fde 2023-04-01 14:55:45,547 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/45b45f0fe50547d29a84f0a9de263fde, entries=150, sequenceid=67, filesize=11.6 K 2023-04-01 14:55:45,548 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 450ms, sequenceid=67, compaction requested=false 2023-04-01 14:55:45,548 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:45,548 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:45,549 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75650069[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:45,551 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:45,552 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,552 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,553 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:45,553 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:45,553 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:45,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:45,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:45,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:45,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:45,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:45,554 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:45,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:45,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:45,554 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:45,554 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:45,557 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:45,558 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,558 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:45,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:45,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:45,558 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:45,560 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:45,561 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:45,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:45,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:45,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:45,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:45,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:45,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,563 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:45,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:45,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:45,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:45,564 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:45,564 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,564 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,564 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:45,564 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:45,564 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:45,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:45,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/e971586a3b24422cad901b3ca1422216 2023-04-01 14:55:45,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361005580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361005581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361005582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361005583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361005585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,664 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:45,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:45,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:45,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:45,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:45,667 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:45,667 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:45,667 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:45,667 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:45,668 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,668 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,669 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:45,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:45,670 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,670 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:45,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:45,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:45,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:45,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:45,671 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:45,673 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,673 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:45,673 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,673 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:45,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:45,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a02abf[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:45,674 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:45,675 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,680 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,680 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:45,680 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:45,680 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:45,680 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,680 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:45,681 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:45,681 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:45,681 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,681 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:45,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:45,683 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:45,684 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:45,684 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:45,684 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:45,684 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:45,685 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,685 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,685 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:45,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:45,686 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:45,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361005686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361005688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,693 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:45,693 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:45,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:45,694 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,694 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,694 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,694 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,694 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,694 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:45,694 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:45,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:45,694 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:45,694 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:45,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:45,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:45,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361005691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361005691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361005694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,794 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:45,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:45,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:45,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:45,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:45,798 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:45,798 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:45,798 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:45,798 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:45,798 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,798 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:45,799 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:45,800 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,800 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:45,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:45,801 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:45,801 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:45,801 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:45,801 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:45,802 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:45,803 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:45,803 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:45,803 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,803 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd67f21[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:45,805 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:45,805 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,807 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:45,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,809 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:45,809 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:45,809 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:45,810 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:45,810 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:45,810 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,810 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:45,813 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:45,813 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:45,813 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:45,813 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:45,813 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:45,814 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,814 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,814 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:45,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:45,815 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,816 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,818 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:45,818 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:45,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:45,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:45,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,819 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:45,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:45,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:45,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,820 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:45,820 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:45,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361005892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361005892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361005897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361005897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:45,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361005899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,920 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:45,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:45,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:45,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:45,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:45,923 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:45,923 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:45,923 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:45,923 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:45,923 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,923 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:45,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:45,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:45,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:45,928 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:45,928 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:45,928 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:45,928 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:45,928 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:45,930 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,930 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:45,931 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,931 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6722396a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:45,932 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:45,932 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,933 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,933 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:45,933 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:45,934 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:45,934 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,934 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:45,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,936 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:45,936 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:45,936 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:45,938 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:45,938 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:45,938 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:45,938 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:45,938 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:45,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:45,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:45,941 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:45,942 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:45,944 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:45,944 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:45,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:45,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:45,944 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,944 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:45,944 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:45,945 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:45,945 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:45,945 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:45,945 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:45,945 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,945 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:45,945 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:45,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:45,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:45,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/ab73bab655ad4ecbbf4531972267ea59 2023-04-01 14:55:46,006 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 14:55:46,045 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:46,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:46,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:46,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:46,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:46,048 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:46,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:46,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:46,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:46,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,048 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:46,051 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:46,051 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,051 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:46,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:46,052 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:46,052 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:46,052 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:46,052 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:46,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,053 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:46,054 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,054 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a990baf[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:46,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:46,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,058 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:46,058 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,058 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:46,059 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,059 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:46,059 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:46,059 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:46,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:46,061 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:46,062 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:46,062 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:46,062 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:46,062 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:46,063 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:46,063 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,063 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:46,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,064 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:46,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:46,067 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:46,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:46,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:46,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:46,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:46,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:46,068 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:46,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:46,068 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:46,168 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:46,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:46,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:46,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:46,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:46,172 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:46,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:46,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:46,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:46,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,177 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:46,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:46,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,177 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:46,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:46,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:46,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:46,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:46,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:46,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:46,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:46,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:46,178 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:46,179 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:46,179 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:46,179 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:46,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:46,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:46,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:46,179 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:46,181 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,181 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:46,181 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:46,181 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b4d2b91[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:46,182 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:46,182 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,183 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,183 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:46,183 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:46,183 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:46,184 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,184 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:46,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,186 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:46,186 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:46,186 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:46,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:46,188 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:46,188 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:46,188 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:46,188 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:46,189 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:46,190 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,190 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,190 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:46,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:46,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,192 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:46,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:46,194 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:46,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:46,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:46,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,195 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:46,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:46,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:46,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,195 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:46,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:46,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361006197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361006197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361006202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361006202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361006205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,295 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:46,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:46,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:46,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:46,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:46,298 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:46,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:46,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:46,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:46,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,299 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:46,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:46,300 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,300 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:46,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:46,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:46,301 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:46,301 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:46,301 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:46,301 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:46,308 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,308 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@770e70ae[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:46,310 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:46,310 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:46,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:46,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:46,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:46,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:46,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:46,312 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:46,315 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:46,315 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:46,315 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:46,315 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:46,315 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:46,316 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,317 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:46,317 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:46,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:46,317 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,318 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:46,322 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:46,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:46,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:46,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:46,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:46,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:46,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,322 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:46,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:46,322 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:46,394 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/bfc4ae97628f478aa76302a54b6a4d3e 2023-04-01 14:55:46,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/e971586a3b24422cad901b3ca1422216 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e971586a3b24422cad901b3ca1422216 2023-04-01 14:55:46,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e971586a3b24422cad901b3ca1422216, entries=200, sequenceid=85, filesize=13.9 K 2023-04-01 14:55:46,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/ab73bab655ad4ecbbf4531972267ea59 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ab73bab655ad4ecbbf4531972267ea59 2023-04-01 14:55:46,405 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ab73bab655ad4ecbbf4531972267ea59, entries=150, sequenceid=85, filesize=11.6 K 2023-04-01 14:55:46,406 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/bfc4ae97628f478aa76302a54b6a4d3e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bfc4ae97628f478aa76302a54b6a4d3e 2023-04-01 14:55:46,409 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bfc4ae97628f478aa76302a54b6a4d3e, entries=150, sequenceid=85, filesize=11.6 K 2023-04-01 14:55:46,410 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 849ms, sequenceid=85, compaction requested=true 2023-04-01 14:55:46,410 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:46,410 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:46,410 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:46,410 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:46,410 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:46,410 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 14:55:46,416 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:46,416 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:55:46,416 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:46,416 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:46,416 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ef2d03f682b548719f35e193f5f8d8cb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/d4256e958bb644d091fc649b540b3719, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ab73bab655ad4ecbbf4531972267ea59] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=34.9 K 2023-04-01 14:55:46,417 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/A is initiating minor compaction (all files) 2023-04-01 14:55:46,417 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/A in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:46,417 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/cd74b407566949008d703f7eec4995e8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7e577242145e42578af6b53d80aa7d31, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e971586a3b24422cad901b3ca1422216] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=37.2 K 2023-04-01 14:55:46,417 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting ef2d03f682b548719f35e193f5f8d8cb, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680360944249 2023-04-01 14:55:46,417 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cd74b407566949008d703f7eec4995e8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680360944249 2023-04-01 14:55:46,419 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d4256e958bb644d091fc649b540b3719, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680360944610 2023-04-01 14:55:46,419 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 7e577242145e42578af6b53d80aa7d31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680360944610 2023-04-01 14:55:46,419 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting ab73bab655ad4ecbbf4531972267ea59, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680360945249 2023-04-01 14:55:46,419 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e971586a3b24422cad901b3ca1422216, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680360945249 2023-04-01 14:55:46,423 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:46,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:46,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:46,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:46,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:46,428 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:46,428 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:46,428 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:46,428 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:46,429 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#A#compaction#414 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:46,429 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#415 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:46,430 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,430 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,433 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:46,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:46,433 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:46,434 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,434 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:46,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:46,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:46,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:46,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:46,440 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:46,452 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/fd22b55cd0fd4d4a98c9464c622ec69c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/fd22b55cd0fd4d4a98c9464c622ec69c 2023-04-01 14:55:46,457 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into fd22b55cd0fd4d4a98c9464c622ec69c(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:46,457 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:46,457 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=13, startTime=1680360946410; duration=0sec 2023-04-01 14:55:46,457 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:46,457 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:46,458 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:46,458 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/C is initiating minor compaction (all files) 2023-04-01 14:55:46,458 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/C in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:46,458 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/6a8131293d624cf0b17a39fc81c8cdce, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/45b45f0fe50547d29a84f0a9de263fde, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bfc4ae97628f478aa76302a54b6a4d3e] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=34.9 K 2023-04-01 14:55:46,458 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 6a8131293d624cf0b17a39fc81c8cdce, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680360944249 2023-04-01 14:55:46,458 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 45b45f0fe50547d29a84f0a9de263fde, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680360944610 2023-04-01 14:55:46,459 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting bfc4ae97628f478aa76302a54b6a4d3e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680360945249 2023-04-01 14:55:46,460 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:46,460 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:46,460 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:46,460 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:46,460 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 14:55:46,460 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:46,460 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:46,460 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:46,461 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:46,461 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:46,461 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:46,464 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#C#compaction#416 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:46,470 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/220b7d59b99f4e7abd8c089a0adae0b9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/220b7d59b99f4e7abd8c089a0adae0b9 2023-04-01 14:55:46,474 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/A of 2ddd6fcff99c4aa857940e9dc8e40f15 into 220b7d59b99f4e7abd8c089a0adae0b9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:46,474 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:46,474 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/A, priority=13, startTime=1680360946410; duration=0sec 2023-04-01 14:55:46,474 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:46,481 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/c836d2954e494f8fac3da7ea6b0d88c0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/c836d2954e494f8fac3da7ea6b0d88c0 2023-04-01 14:55:46,486 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/C of 2ddd6fcff99c4aa857940e9dc8e40f15 into c836d2954e494f8fac3da7ea6b0d88c0(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:46,486 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:46,486 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/C, priority=13, startTime=1680360946410; duration=0sec 2023-04-01 14:55:46,486 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:46,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:46,703 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:46,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361006714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361006715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361006717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361006718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361006719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361006820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361006820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361006824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361006824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361006824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:46,875 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/22beee555e2d4cb8ae7be231e0113544 2023-04-01 14:55:47,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361007026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361007026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361007027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361007028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361007028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,283 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/275b1558f73e43e0adeeb77357903617 2023-04-01 14:55:47,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361007331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361007332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361007333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361007333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361007334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,697 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/8087e2f8d0164b15b774c1684d0c1a78 2023-04-01 14:55:47,700 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/22beee555e2d4cb8ae7be231e0113544 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/22beee555e2d4cb8ae7be231e0113544 2023-04-01 14:55:47,704 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/22beee555e2d4cb8ae7be231e0113544, entries=150, sequenceid=103, filesize=11.6 K 2023-04-01 14:55:47,704 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/275b1558f73e43e0adeeb77357903617 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/275b1558f73e43e0adeeb77357903617 2023-04-01 14:55:47,707 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/275b1558f73e43e0adeeb77357903617, entries=150, sequenceid=103, filesize=11.6 K 2023-04-01 14:55:47,708 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/8087e2f8d0164b15b774c1684d0c1a78 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8087e2f8d0164b15b774c1684d0c1a78 2023-04-01 14:55:47,711 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8087e2f8d0164b15b774c1684d0c1a78, entries=150, sequenceid=103, filesize=11.6 K 2023-04-01 14:55:47,711 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 1251ms, sequenceid=103, compaction requested=false 2023-04-01 14:55:47,712 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:47,712 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:47,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:47,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:47,713 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:47,714 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:47,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:47,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@111fe4c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:47,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:47,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:47,716 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:47,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:47,716 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:47,716 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:47,716 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:47,716 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,717 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:47,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:47,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:47,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:47,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:47,722 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:47,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:47,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:47,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:47,722 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:47,723 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:47,723 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:47,723 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:47,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:47,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:47,724 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,725 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:47,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:47,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:47,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:47,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:47,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:47,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:47,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:47,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:47,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:47,728 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:47,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:47,729 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:47,729 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:47,829 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:47,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:47,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:47,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:47,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:47,832 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:47,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:47,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:47,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:47,833 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:47,833 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:47,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,834 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:47,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:47,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:47,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:47,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:47,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:47,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:47,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:47,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:47,835 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:47,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:47,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:47,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:47,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:47,836 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 14:55:47,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:47,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:47,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:47,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:47,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:47,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:47,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:47,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:47,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361007853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361007854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361007856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361007857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361007858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361007967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361007967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361007967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361007968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:47,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:47,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361007968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361008173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361008173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361008173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361008173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361008174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,253 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/544e93c2f6a0459fa91a9f3778efcddf 2023-04-01 14:55:48,278 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/1c1d1142939d473c8b4199f1ac2f085c 2023-04-01 14:55:48,314 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/bf813067646a441883087a590fa11ebf 2023-04-01 14:55:48,318 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/544e93c2f6a0459fa91a9f3778efcddf as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/544e93c2f6a0459fa91a9f3778efcddf 2023-04-01 14:55:48,325 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/544e93c2f6a0459fa91a9f3778efcddf, entries=150, sequenceid=120, filesize=11.6 K 2023-04-01 14:55:48,326 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/1c1d1142939d473c8b4199f1ac2f085c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1c1d1142939d473c8b4199f1ac2f085c 2023-04-01 14:55:48,332 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1c1d1142939d473c8b4199f1ac2f085c, entries=150, sequenceid=120, filesize=11.6 K 2023-04-01 14:55:48,333 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/bf813067646a441883087a590fa11ebf as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf813067646a441883087a590fa11ebf 2023-04-01 14:55:48,336 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf813067646a441883087a590fa11ebf, entries=150, sequenceid=120, filesize=11.6 K 2023-04-01 14:55:48,337 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 501ms, sequenceid=120, compaction requested=true 2023-04-01 14:55:48,337 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:48,337 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:48,337 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:48,337 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:48,337 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,337 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:48,338 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:48,342 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732759b2[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:48,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:48,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,348 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:48,348 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,348 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:48,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,349 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:48,349 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,349 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:48,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:48,352 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:48,352 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:48,352 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:48,352 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:48,352 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:48,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,356 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:48,356 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,357 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:48,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:48,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:48,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:48,361 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,361 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,361 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,361 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:48,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:48,361 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,361 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:48,361 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,361 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,361 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:48,362 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:48,462 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:48,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:48,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:48,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:48,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:48,468 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:48,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:48,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:48,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:48,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,469 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,469 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:48,470 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:48,470 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,470 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:48,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:48,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:48,472 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:48,472 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:48,472 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:48,472 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 14:55:48,472 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:48,472 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:48,472 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:48,472 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:48,472 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:48,472 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:48,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:48,485 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:48,507 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/5f99100790ab4c13b44647cad6f4ecae 2023-04-01 14:55:48,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361008498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361008500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361008505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361008505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361008508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,522 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/207e3055252c4446a73bd52a9fcaa828 2023-04-01 14:55:48,533 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/8524f835b0184788856844116815c4a4 2023-04-01 14:55:48,537 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/5f99100790ab4c13b44647cad6f4ecae as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5f99100790ab4c13b44647cad6f4ecae 2023-04-01 14:55:48,540 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5f99100790ab4c13b44647cad6f4ecae, entries=150, sequenceid=137, filesize=11.8 K 2023-04-01 14:55:48,541 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/207e3055252c4446a73bd52a9fcaa828 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/207e3055252c4446a73bd52a9fcaa828 2023-04-01 14:55:48,544 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/207e3055252c4446a73bd52a9fcaa828, entries=150, sequenceid=137, filesize=11.8 K 2023-04-01 14:55:48,545 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/8524f835b0184788856844116815c4a4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8524f835b0184788856844116815c4a4 2023-04-01 14:55:48,549 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8524f835b0184788856844116815c4a4, entries=150, sequenceid=137, filesize=11.8 K 2023-04-01 14:55:48,550 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 78ms, sequenceid=137, compaction requested=true 2023-04-01 14:55:48,550 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:48,550 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:48,552 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:48,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@144e5e4[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:48,553 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:48,553 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,556 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:48,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:48,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:48,556 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,556 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:48,557 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,557 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:48,557 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:48,557 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:48,559 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:48,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:48,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:48,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:48,559 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:48,561 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,561 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,561 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:48,562 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,563 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,565 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,565 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,565 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,565 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,565 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,565 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,566 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,565 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:48,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:48,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:48,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:48,566 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:48,566 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:48,566 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:48,566 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:48,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:48,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:48,614 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 14:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:48,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/632675a61a444c72a87b598725d468d7 2023-04-01 14:55:48,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/286bab0c821b446cad2f1a3d1400aeff 2023-04-01 14:55:48,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361008628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361008631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361008635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361008636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361008637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,666 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:48,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:48,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:48,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:48,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:48,669 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:48,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:48,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:48,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:48,670 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,670 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,676 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,676 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:48,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:48,676 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,676 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:48,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:48,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:48,680 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:48,680 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:48,680 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:48,681 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:48,682 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,683 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,683 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb6e62[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:48,685 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:48,685 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,687 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:48,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:48,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:48,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:48,688 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:48,688 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:48,688 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:48,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:48,693 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:48,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:48,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:48,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:48,693 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:48,696 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,696 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,696 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:48,697 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,700 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,703 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:48,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:48,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:48,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:48,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:48,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:48,703 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:48,703 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:48,703 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:48,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:48,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361008738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361008738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361008744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361008747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361008747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,804 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:48,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:48,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:48,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:48,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:48,809 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:48,809 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:48,809 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:48,809 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:48,809 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,809 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,810 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:48,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:48,811 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,811 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,811 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:48,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:48,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:48,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:48,825 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:48,825 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:48,825 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:48,834 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:48,834 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:48,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:48,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:48,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:48,836 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:48,838 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,838 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,839 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b261309[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:48,840 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:48,841 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,844 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:48,844 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:48,844 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:48,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:48,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:48,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:48,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:48,849 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:48,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:48,852 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:48,852 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:48,852 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:48,852 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:48,852 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:48,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:48,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,855 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,856 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:48,861 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:48,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:48,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:48,861 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:48,861 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:48,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:48,861 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:48,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:48,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361008946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361008947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361008951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361008957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:48,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361008958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,961 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:48,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:48,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:48,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:48,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:48,964 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:48,964 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:48,964 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:48,965 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:48,965 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,965 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,966 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:48,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:48,968 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,968 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,968 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:48,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:48,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:48,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:48,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:48,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:48,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:48,970 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:48,970 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:48,970 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:48,970 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:48,970 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:48,970 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:48,970 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:48,970 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,970 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:48,970 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:48,973 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,973 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,973 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,973 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:48,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42853116[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:48,975 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:48,975 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,976 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:48,976 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,976 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:48,978 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,978 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:48,978 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,978 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:48,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:48,983 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:48,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:48,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:48,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:48,983 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:48,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:48,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:48,987 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,988 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,997 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,997 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:48,997 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,997 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:48,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:48,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:48,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:48,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:48,997 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:48,997 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:48,997 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:48,997 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,997 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:48,997 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:48,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:48,998 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:48,998 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:49,045 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/58b4b7c0a0b944c49c17bd82f5e61bf6 2023-04-01 14:55:49,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/632675a61a444c72a87b598725d468d7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/632675a61a444c72a87b598725d468d7 2023-04-01 14:55:49,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/632675a61a444c72a87b598725d468d7, entries=200, sequenceid=153, filesize=14.1 K 2023-04-01 14:55:49,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/286bab0c821b446cad2f1a3d1400aeff as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/286bab0c821b446cad2f1a3d1400aeff 2023-04-01 14:55:49,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/286bab0c821b446cad2f1a3d1400aeff, entries=150, sequenceid=153, filesize=11.8 K 2023-04-01 14:55:49,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/58b4b7c0a0b944c49c17bd82f5e61bf6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/58b4b7c0a0b944c49c17bd82f5e61bf6 2023-04-01 14:55:49,061 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/58b4b7c0a0b944c49c17bd82f5e61bf6, entries=150, sequenceid=153, filesize=11.8 K 2023-04-01 14:55:49,062 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 448ms, sequenceid=153, compaction requested=true 2023-04-01 14:55:49,062 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:49,062 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:49,062 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:49,062 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:55:49,062 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:55:49,062 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:49,063 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62326 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:55:49,064 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59936 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:55:49,064 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/A is initiating minor compaction (all files) 2023-04-01 14:55:49,064 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:55:49,064 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/A in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,064 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,064 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/220b7d59b99f4e7abd8c089a0adae0b9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/22beee555e2d4cb8ae7be231e0113544, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/544e93c2f6a0459fa91a9f3778efcddf, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5f99100790ab4c13b44647cad6f4ecae, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/632675a61a444c72a87b598725d468d7] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=60.9 K 2023-04-01 14:55:49,064 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/fd22b55cd0fd4d4a98c9464c622ec69c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/275b1558f73e43e0adeeb77357903617, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1c1d1142939d473c8b4199f1ac2f085c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/207e3055252c4446a73bd52a9fcaa828, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/286bab0c821b446cad2f1a3d1400aeff] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=58.5 K 2023-04-01 14:55:49,064 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 220b7d59b99f4e7abd8c089a0adae0b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680360945249 2023-04-01 14:55:49,064 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting fd22b55cd0fd4d4a98c9464c622ec69c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680360945249 2023-04-01 14:55:49,064 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 22beee555e2d4cb8ae7be231e0113544, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680360945564 2023-04-01 14:55:49,065 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 275b1558f73e43e0adeeb77357903617, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680360945564 2023-04-01 14:55:49,065 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 544e93c2f6a0459fa91a9f3778efcddf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680360946711 2023-04-01 14:55:49,065 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1c1d1142939d473c8b4199f1ac2f085c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680360946711 2023-04-01 14:55:49,065 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 5f99100790ab4c13b44647cad6f4ecae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680360947849 2023-04-01 14:55:49,065 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 207e3055252c4446a73bd52a9fcaa828, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680360947849 2023-04-01 14:55:49,065 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 632675a61a444c72a87b598725d468d7, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680360948498 2023-04-01 14:55:49,066 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 286bab0c821b446cad2f1a3d1400aeff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680360948498 2023-04-01 14:55:49,077 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#A#compaction#429 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:49,078 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#430 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:49,096 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/239b89c38ac3478b8e147e43c56ba919 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/239b89c38ac3478b8e147e43c56ba919 2023-04-01 14:55:49,098 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:49,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:49,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:49,100 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into 239b89c38ac3478b8e147e43c56ba919(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:49,100 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:49,100 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=11, startTime=1680360949062; duration=0sec 2023-04-01 14:55:49,100 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:49,100 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:55:49,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:49,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:49,101 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:49,101 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,101 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:49,101 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:49,101 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,102 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,102 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59936 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:55:49,102 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/C is initiating minor compaction (all files) 2023-04-01 14:55:49,102 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/C in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,102 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/c836d2954e494f8fac3da7ea6b0d88c0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8087e2f8d0164b15b774c1684d0c1a78, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf813067646a441883087a590fa11ebf, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8524f835b0184788856844116815c4a4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/58b4b7c0a0b944c49c17bd82f5e61bf6] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=58.5 K 2023-04-01 14:55:49,102 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c836d2954e494f8fac3da7ea6b0d88c0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680360945249 2023-04-01 14:55:49,102 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 8087e2f8d0164b15b774c1684d0c1a78, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680360945564 2023-04-01 14:55:49,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:49,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:49,103 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting bf813067646a441883087a590fa11ebf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680360946711 2023-04-01 14:55:49,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,103 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 8524f835b0184788856844116815c4a4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680360947849 2023-04-01 14:55:49,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,104 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 58b4b7c0a0b944c49c17bd82f5e61bf6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680360948498 2023-04-01 14:55:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:49,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:49,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:49,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:49,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:49,105 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,105 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:49,105 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 14:55:49,105 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:49,105 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:49,105 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:49,105 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:49,105 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:49,105 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:49,111 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/6243b27b51f54a0994a6c481735ad7c1 2023-04-01 14:55:49,112 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#C#compaction#432 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:49,120 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/b0e674757cbf40f894f6c05f9586a4c9 2023-04-01 14:55:49,127 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/7e097be164d74f1aa71294ec29a3c316 2023-04-01 14:55:49,130 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/6243b27b51f54a0994a6c481735ad7c1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6243b27b51f54a0994a6c481735ad7c1 2023-04-01 14:55:49,134 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6243b27b51f54a0994a6c481735ad7c1, entries=150, sequenceid=170, filesize=11.8 K 2023-04-01 14:55:49,134 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/b0e674757cbf40f894f6c05f9586a4c9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b0e674757cbf40f894f6c05f9586a4c9 2023-04-01 14:55:49,141 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b0e674757cbf40f894f6c05f9586a4c9, entries=150, sequenceid=170, filesize=11.8 K 2023-04-01 14:55:49,141 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/7e097be164d74f1aa71294ec29a3c316 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7e097be164d74f1aa71294ec29a3c316 2023-04-01 14:55:49,150 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7e097be164d74f1aa71294ec29a3c316, entries=150, sequenceid=170, filesize=11.8 K 2023-04-01 14:55:49,150 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 45ms, sequenceid=170, compaction requested=false 2023-04-01 14:55:49,150 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:49,150 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,150 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:49,150 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:49,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:49,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:49,153 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,153 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,153 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,153 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:49,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64c89767[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:49,155 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:49,155 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,157 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:49,157 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,157 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:49,159 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,159 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:49,159 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,159 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:49,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:49,162 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:49,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:49,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:49,163 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:49,163 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:49,164 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,164 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,165 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,165 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,166 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,168 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,168 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:49,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:49,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:49,168 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,168 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:49,168 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:49,168 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:49,169 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,169 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,169 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,169 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:49,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:49,169 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:49,169 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:49,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:49,266 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:55:49,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:49,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:49,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:49,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:49,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:49,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:49,269 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:49,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:49,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:49,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:49,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:49,273 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:49,273 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/b189447fd8ed40dfb8fd0873364365ab 2023-04-01 14:55:49,273 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:49,273 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:49,274 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,274 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,275 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:49,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:49,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,275 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,276 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:49,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:49,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:49,277 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,277 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:49,277 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:49,277 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:49,284 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,284 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,284 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/2f9b695b4b574049b4b433ca5bb0284a 2023-04-01 14:55:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,285 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,285 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:49,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a2e5de2[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:49,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:49,287 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,288 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:49,288 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:49,288 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:49,288 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,288 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:49,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:49,289 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:49,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:49,292 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:49,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:49,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:49,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:49,292 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:49,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,301 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:49,301 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,301 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:49,301 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:49,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:49,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:49,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:49,301 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:49,302 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:49,302 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:49,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680360949280, max timestamp=1680360949303Num uniques 859; 2023-04-01 14:55:49,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-01 14:55:49,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680360949280, max timestamp=1680360949303Num uniques 859; 2023-04-01 14:55:49,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-01 14:55:49,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680360949280, max timestamp=1680360949303Num uniques 859; 2023-04-01 14:55:49,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-01 14:55:49,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361009326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361009327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361009330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361009330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361009333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,402 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:49,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:49,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:49,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:49,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:49,416 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:49,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:49,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:49,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:49,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:49,419 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:49,420 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,420 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:49,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:49,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,424 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:49,430 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:49,430 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:49,430 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,431 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:49,431 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:49,431 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:49,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:49,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:49,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:49,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,432 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,432 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:49,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1abbff5[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:49,435 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:49,435 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,437 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,437 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:49,437 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:49,438 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,438 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:49,439 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:49,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:49,442 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:49,442 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:49,442 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:49,442 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:49,442 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:49,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361009434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:49,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,444 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,445 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,448 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:49,448 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:49,448 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,449 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,449 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:49,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:49,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:49,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361009438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,449 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,449 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,449 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:49,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:49,449 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361009439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,449 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:49,450 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:49,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:49,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361009440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361009440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,491 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/3043b40fb6764642bbe3bbaac4629783 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3043b40fb6764642bbe3bbaac4629783 2023-04-01 14:55:49,498 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/A of 2ddd6fcff99c4aa857940e9dc8e40f15 into 3043b40fb6764642bbe3bbaac4629783(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:49,499 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:49,499 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/A, priority=11, startTime=1680360949062; duration=0sec 2023-04-01 14:55:49,499 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:49,526 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/ccd61d6b54f64e4997b1b4c97c8bf290 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ccd61d6b54f64e4997b1b4c97c8bf290 2023-04-01 14:55:49,533 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/C of 2ddd6fcff99c4aa857940e9dc8e40f15 into ccd61d6b54f64e4997b1b4c97c8bf290(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:49,534 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:49,534 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/C, priority=11, startTime=1680360949062; duration=0sec 2023-04-01 14:55:49,534 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:49,550 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:49,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:49,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:49,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:49,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:49,555 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:49,555 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,555 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:49,555 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:49,555 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,555 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,560 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:49,560 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,561 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,561 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:49,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:49,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:49,562 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,562 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:49,562 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:49,562 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:49,563 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,563 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:49,563 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:49,564 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,564 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,567 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6acd3357[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:49,568 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:49,568 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,570 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,570 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:49,570 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:49,570 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,570 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:49,570 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,571 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:49,571 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,571 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:49,575 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:49,576 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:49,576 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:49,576 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:49,576 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:49,578 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,579 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,579 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:49,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,581 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,583 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:49,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:49,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:49,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:49,586 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:49,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:49,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:49,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:49,587 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:49,587 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:49,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361009645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361009650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361009651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361009651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361009652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,687 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:49,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:49,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:49,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:49,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:49,690 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:49,690 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,690 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:49,690 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:49,691 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,691 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,692 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:49,692 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:49,692 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,692 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:49,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:49,693 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,693 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:49,694 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:49,694 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:49,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/b91ecc87b3444541af0f3422c832eb62 2023-04-01 14:55:49,695 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,695 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:49,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a9ee2c6[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:49,697 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:49,697 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/b189447fd8ed40dfb8fd0873364365ab as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/b189447fd8ed40dfb8fd0873364365ab 2023-04-01 14:55:49,700 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:49,703 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/b189447fd8ed40dfb8fd0873364365ab, entries=350, sequenceid=180, filesize=21.1 K 2023-04-01 14:55:49,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/2f9b695b4b574049b4b433ca5bb0284a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2f9b695b4b574049b4b433ca5bb0284a 2023-04-01 14:55:49,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:49,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:49,705 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:49,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2f9b695b4b574049b4b433ca5bb0284a, entries=150, sequenceid=180, filesize=11.8 K 2023-04-01 14:55:49,708 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/b91ecc87b3444541af0f3422c832eb62 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b91ecc87b3444541af0f3422c832eb62 2023-04-01 14:55:49,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:49,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:49,712 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:49,712 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:49,713 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:49,713 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:49,713 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:49,714 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,714 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:49,714 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:49,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:49,715 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,716 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:49,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:49,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:49,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,721 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:49,722 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:49,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:49,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:49,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:49,722 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:49,722 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:49,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:49,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b91ecc87b3444541af0f3422c832eb62, entries=150, sequenceid=180, filesize=11.8 K 2023-04-01 14:55:49,725 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 459ms, sequenceid=180, compaction requested=true 2023-04-01 14:55:49,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:49,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:49,728 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:49,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:49,728 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:49,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:49,729 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 46048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:49,729 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/A is initiating minor compaction (all files) 2023-04-01 14:55:49,729 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/A in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,729 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3043b40fb6764642bbe3bbaac4629783, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6243b27b51f54a0994a6c481735ad7c1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/b189447fd8ed40dfb8fd0873364365ab] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=45.0 K 2023-04-01 14:55:49,732 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:49,732 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:55:49,732 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,732 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/239b89c38ac3478b8e147e43c56ba919, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b0e674757cbf40f894f6c05f9586a4c9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2f9b695b4b574049b4b433ca5bb0284a] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=35.6 K 2023-04-01 14:55:49,732 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 3043b40fb6764642bbe3bbaac4629783, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680360948498 2023-04-01 14:55:49,733 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 239b89c38ac3478b8e147e43c56ba919, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680360948498 2023-04-01 14:55:49,734 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 6243b27b51f54a0994a6c481735ad7c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680360948630 2023-04-01 14:55:49,734 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b0e674757cbf40f894f6c05f9586a4c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680360948630 2023-04-01 14:55:49,734 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2f9b695b4b574049b4b433ca5bb0284a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680360949263 2023-04-01 14:55:49,735 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting b189447fd8ed40dfb8fd0873364365ab, keycount=350, bloomtype=ROW, size=21.1 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680360949257 2023-04-01 14:55:49,748 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#438 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:49,750 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#A#compaction#439 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:49,782 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/177adc8a122e4bb2bae4269614579c6e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/177adc8a122e4bb2bae4269614579c6e 2023-04-01 14:55:49,786 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into 177adc8a122e4bb2bae4269614579c6e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:49,787 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:49,787 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=13, startTime=1680360949728; duration=0sec 2023-04-01 14:55:49,787 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:49,787 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:49,788 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:49,788 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/C is initiating minor compaction (all files) 2023-04-01 14:55:49,788 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/C in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,788 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ccd61d6b54f64e4997b1b4c97c8bf290, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7e097be164d74f1aa71294ec29a3c316, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b91ecc87b3444541af0f3422c832eb62] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=35.6 K 2023-04-01 14:55:49,789 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ccd61d6b54f64e4997b1b4c97c8bf290, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1680360948498 2023-04-01 14:55:49,789 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 7e097be164d74f1aa71294ec29a3c316, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680360948630 2023-04-01 14:55:49,790 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b91ecc87b3444541af0f3422c832eb62, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680360949263 2023-04-01 14:55:49,793 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/57814530ced6486093bccae7f7874f36 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/57814530ced6486093bccae7f7874f36 2023-04-01 14:55:49,799 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/A of 2ddd6fcff99c4aa857940e9dc8e40f15 into 57814530ced6486093bccae7f7874f36(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:49,799 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:49,799 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/A, priority=13, startTime=1680360949728; duration=0sec 2023-04-01 14:55:49,799 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:49,801 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#C#compaction#440 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:49,822 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:49,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:49,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:49,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:49,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:49,825 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:49,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:49,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:49,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:49,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:49,827 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,828 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,828 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:49,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:49,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:49,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:49,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:49,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:49,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:49,848 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:49,848 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:49,848 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-04-01 14:55:49,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:49,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:55:49,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:49,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:55:49,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:49,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:55:49,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:49,960 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:49,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361009967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361009968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361009970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361009970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:49,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:49,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361009972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361010074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361010076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361010077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361010077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361010081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,222 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/cfdafa2650484cdebd8bbae2c84a9a48 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/cfdafa2650484cdebd8bbae2c84a9a48 2023-04-01 14:55:50,226 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/C of 2ddd6fcff99c4aa857940e9dc8e40f15 into cfdafa2650484cdebd8bbae2c84a9a48(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:50,227 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:50,227 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/C, priority=13, startTime=1680360949728; duration=0sec 2023-04-01 14:55:50,227 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:50,264 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/2957beb01ba543c09268d363cb15843b 2023-04-01 14:55:50,276 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/2414ea28cc5045e3a95c1b9d27fe9e7f 2023-04-01 14:55:50,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361010280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361010281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361010281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361010281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361010287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,304 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/86b4bd188a7f40f6b30691ed3d1fad8a 2023-04-01 14:55:50,311 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/2957beb01ba543c09268d363cb15843b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2957beb01ba543c09268d363cb15843b 2023-04-01 14:55:50,314 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2957beb01ba543c09268d363cb15843b, entries=150, sequenceid=218, filesize=11.8 K 2023-04-01 14:55:50,314 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/2414ea28cc5045e3a95c1b9d27fe9e7f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2414ea28cc5045e3a95c1b9d27fe9e7f 2023-04-01 14:55:50,318 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2414ea28cc5045e3a95c1b9d27fe9e7f, entries=150, sequenceid=218, filesize=11.8 K 2023-04-01 14:55:50,319 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/86b4bd188a7f40f6b30691ed3d1fad8a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/86b4bd188a7f40f6b30691ed3d1fad8a 2023-04-01 14:55:50,323 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/86b4bd188a7f40f6b30691ed3d1fad8a, entries=150, sequenceid=218, filesize=11.8 K 2023-04-01 14:55:50,324 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=46.96 KB/48090 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 476ms, sequenceid=218, compaction requested=false 2023-04-01 14:55:50,324 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:50,324 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:50,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,326 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:50,326 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,327 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:50,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:50,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217673a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:50,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:50,331 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,332 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:50,332 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,332 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:50,335 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:50,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:50,337 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:50,337 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:50,337 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:50,337 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:50,337 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:50,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:50,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,340 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:50,340 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,342 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:50,345 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:50,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:50,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:50,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:50,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:50,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:50,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:50,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:50,345 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:50,346 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:50,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:50,446 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:50,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:50,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:50,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:50,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:50,449 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:50,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:50,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:50,449 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:50,452 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,452 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:50,453 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,453 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:50,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:50,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:50,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:50,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:50,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:50,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:50,455 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:50,455 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:50,456 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:50,456 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:55:50,456 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:50,456 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:50,456 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:50,456 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:50,456 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:50,456 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:50,476 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/7a8c7e62321048d9997d7a0a88ad065d 2023-04-01 14:55:50,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:50,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:50,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361010611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360950601, max timestamp=1680360950613Num uniques 859; 2023-04-01 14:55:50,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:55:50,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360950601, max timestamp=1680360950613Num uniques 859; 2023-04-01 14:55:50,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360950601, max timestamp=1680360950613Num uniques 859; 2023-04-01 14:55:50,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:55:50,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:55:50,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361010639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361010643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361010644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361010651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361010716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361010752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361010752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361010755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361010755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,906 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/5348a97b3fee46d8b92f3cb7a288b6de 2023-04-01 14:55:50,915 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/18fb72ac835644aebf0e158434eeeb59 2023-04-01 14:55:50,921 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/7a8c7e62321048d9997d7a0a88ad065d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7a8c7e62321048d9997d7a0a88ad065d 2023-04-01 14:55:50,923 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7a8c7e62321048d9997d7a0a88ad065d, entries=150, sequenceid=229, filesize=11.8 K 2023-04-01 14:55:50,924 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/5348a97b3fee46d8b92f3cb7a288b6de as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5348a97b3fee46d8b92f3cb7a288b6de 2023-04-01 14:55:50,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361010924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,927 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5348a97b3fee46d8b92f3cb7a288b6de, entries=150, sequenceid=229, filesize=11.8 K 2023-04-01 14:55:50,928 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/18fb72ac835644aebf0e158434eeeb59 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/18fb72ac835644aebf0e158434eeeb59 2023-04-01 14:55:50,931 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/18fb72ac835644aebf0e158434eeeb59, entries=150, sequenceid=229, filesize=11.8 K 2023-04-01 14:55:50,932 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 476ms, sequenceid=229, compaction requested=true 2023-04-01 14:55:50,932 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:50,932 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:50,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,937 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,938 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:50,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:50,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e1f249a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:50,939 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:50,939 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,940 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:50,940 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:50,940 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:50,940 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,941 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:50,942 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,942 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:50,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:50,946 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:50,946 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:50,946 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:50,946 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:50,946 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:50,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:50,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:50,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:50,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:50,952 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:50,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:50,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:50,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:50,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:50,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:50,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:50,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:50,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:50,953 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:50,953 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:50,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:50,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:50,961 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-01 14:55:50,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:50,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 14:55:50,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680360950636, max timestamp=1680360950960Num uniques 1018; 2023-04-01 14:55:50,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-01 14:55:50,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:50,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-01 14:55:50,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:50,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:55:50,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361010977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361010979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361010980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:50,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:50,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361010981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,053 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:51,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:51,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:51,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:51,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:51,056 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:51,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:51,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:51,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,056 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,057 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,057 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,058 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,058 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:51,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:51,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:51,059 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,059 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:51,059 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:51,059 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:51,061 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,061 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,061 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,061 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:51,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:51,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d4bf385[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:51,063 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:51,063 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,063 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:51,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:51,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:51,069 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:51,069 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:51,069 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:51,069 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:51,069 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:51,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,071 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,073 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,075 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,075 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,075 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,075 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,075 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,075 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:51,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:51,075 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:51,075 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,075 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:51,075 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,075 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:51,075 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361011082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361011088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361011088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361011088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,175 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:51,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:51,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:51,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:51,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:51,178 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:51,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:51,178 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:51,179 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,179 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,182 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,182 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,183 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,183 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:51,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:51,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:51,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:51,184 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,184 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:51,184 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:51,184 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:51,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@607ea238[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:51,188 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:51,188 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:51,190 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,190 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,191 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:51,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:51,193 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:51,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:51,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:51,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:51,193 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:51,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,195 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,197 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,200 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,200 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,200 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,200 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:51,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:51,200 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,200 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:51,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,200 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,200 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,200 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:51,200 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,200 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,200 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361011229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,302 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:51,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:51,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:51,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:51,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:51,305 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:51,305 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,305 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:51,305 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:51,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361011289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,307 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361011303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:51,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361011303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361011303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:51,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:51,309 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,309 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:51,309 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:51,309 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:51,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e927ba[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:51,312 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:51,312 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,313 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,314 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:51,314 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:51,314 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:51,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,314 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:51,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,316 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:51,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:51,317 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:51,318 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:51,318 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:51,318 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:51,318 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:51,319 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,319 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,319 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,320 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,321 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,323 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,323 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,323 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:51,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:51,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,324 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,324 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:51,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,324 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:51,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/2dc4bd2f94e34f0e932b8d8416205de4 2023-04-01 14:55:51,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/ded4aaa683e6491faba333c1de387285 2023-04-01 14:55:51,424 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:51,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:51,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:51,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:51,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:51,427 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:51,427 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,427 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:51,427 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:51,428 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,428 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,429 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,429 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,429 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,429 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:51,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:51,430 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:51,430 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,430 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:51,431 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,431 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:51,431 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:51,431 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:51,433 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,433 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,434 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,434 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:51,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e75e3e9[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:51,435 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:51,435 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,436 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:51,436 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,436 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:51,436 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:51,438 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,438 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:51,438 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,438 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:51,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:51,440 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:51,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:51,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:51,440 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:51,440 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:51,441 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,442 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,443 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,446 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,446 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,446 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,446 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:51,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:51,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,446 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:51,447 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:51,446 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,447 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:51,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,447 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,547 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:51,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:51,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:51,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:51,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:51,550 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:51,550 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,550 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:51,550 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:51,551 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,551 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,552 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,552 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,552 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,552 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:51,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:51,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:51,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:51,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:51,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:51,554 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,554 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:51,555 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:51,555 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:51,559 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,559 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,559 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,559 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:51,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca0f079[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:51,561 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:51,561 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,563 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:51,563 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,563 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:51,563 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:51,563 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:51,564 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:51,564 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,564 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,564 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:51,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:51,566 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:51,567 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:51,567 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:51,567 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:51,567 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:51,568 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,568 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,569 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,569 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,573 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,573 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,573 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,573 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,573 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:51,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:51,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,574 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,574 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:51,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361011607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48032 deadline: 1680361011608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48036 deadline: 1680361011610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361011610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,674 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:51,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:51,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:51,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:51,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:51,677 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:51,677 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,677 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:51,678 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:51,678 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,678 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,680 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,680 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,681 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,681 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:51,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:51,682 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,682 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:51,682 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:51,682 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:51,684 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69dae2b7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:51,686 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:51,686 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,687 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:51,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:51,690 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,690 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:51,693 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:51,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:51,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:51,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:51,693 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:51,696 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,696 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:51,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,696 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:51,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,696 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,699 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,701 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,701 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,701 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,701 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,701 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,701 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:51,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:51,701 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,701 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:51,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:51,701 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:51,701 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:51,701 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:51,701 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:51,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:51,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361011736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/23fbfc9d9acf4bf3a78b8da7f1a39533 2023-04-01 14:55:51,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/2dc4bd2f94e34f0e932b8d8416205de4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2dc4bd2f94e34f0e932b8d8416205de4 2023-04-01 14:55:51,797 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2dc4bd2f94e34f0e932b8d8416205de4, entries=250, sequenceid=268, filesize=16.7 K 2023-04-01 14:55:51,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/ded4aaa683e6491faba333c1de387285 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ded4aaa683e6491faba333c1de387285 2023-04-01 14:55:51,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ded4aaa683e6491faba333c1de387285, entries=150, sequenceid=268, filesize=11.9 K 2023-04-01 14:55:51,801 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/23fbfc9d9acf4bf3a78b8da7f1a39533 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/23fbfc9d9acf4bf3a78b8da7f1a39533 2023-04-01 14:55:51,801 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:51,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:51,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:51,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:51,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:51,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/23fbfc9d9acf4bf3a78b8da7f1a39533, entries=150, sequenceid=268, filesize=11.9 K 2023-04-01 14:55:51,806 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:51,806 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,806 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:51,806 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:51,806 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~109.58 KB/112210, heapSize ~312.08 KB/319568, currentSize=67.09 KB/68700 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 845ms, sequenceid=268, compaction requested=true 2023-04-01 14:55:51,806 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,806 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:51,806 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:51,807 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:55:51,807 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:55:51,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:51,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:51,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:51,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:51,808 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53652 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:55:51,808 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/A is initiating minor compaction (all files) 2023-04-01 14:55:51,808 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/A in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,808 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48772 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,808 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:55:51,808 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/57814530ced6486093bccae7f7874f36, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2957beb01ba543c09268d363cb15843b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7a8c7e62321048d9997d7a0a88ad065d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2dc4bd2f94e34f0e932b8d8416205de4] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=52.4 K 2023-04-01 14:55:51,809 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,808 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:51,809 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:51,809 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/177adc8a122e4bb2bae4269614579c6e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2414ea28cc5045e3a95c1b9d27fe9e7f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5348a97b3fee46d8b92f3cb7a288b6de, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ded4aaa683e6491faba333c1de387285] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=47.6 K 2023-04-01 14:55:51,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:51,809 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 57814530ced6486093bccae7f7874f36, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680360949263 2023-04-01 14:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:51,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:51,809 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 2957beb01ba543c09268d363cb15843b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680360949329 2023-04-01 14:55:51,809 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 177adc8a122e4bb2bae4269614579c6e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680360949263 2023-04-01 14:55:51,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:51,810 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:51,810 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:51,810 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 7a8c7e62321048d9997d7a0a88ad065d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680360949967 2023-04-01 14:55:51,810 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2414ea28cc5045e3a95c1b9d27fe9e7f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680360949329 2023-04-01 14:55:51,810 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:51,810 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 2dc4bd2f94e34f0e932b8d8416205de4, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680360950636 2023-04-01 14:55:51,810 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:51,810 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:51,810 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,810 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 5348a97b3fee46d8b92f3cb7a288b6de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680360949967 2023-04-01 14:55:51,810 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:51,810 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 14:55:51,810 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ded4aaa683e6491faba333c1de387285, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680360950636 2023-04-01 14:55:51,811 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:51,811 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:51,811 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:51,811 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:51,811 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:51,811 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:51,822 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#451 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:51,822 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#A#compaction#452 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:51,831 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/044d8ec440d542c9a380c5dba51a99cb 2023-04-01 14:55:51,852 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/0d4d8ad2a2fb4426814e51b4faaf069c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/0d4d8ad2a2fb4426814e51b4faaf069c 2023-04-01 14:55:51,853 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/7b5d598e67b64916a4dcc2bf36e1fca4 2023-04-01 14:55:51,856 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/A of 2ddd6fcff99c4aa857940e9dc8e40f15 into 0d4d8ad2a2fb4426814e51b4faaf069c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:51,856 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:51,856 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/A, priority=12, startTime=1680360951807; duration=0sec 2023-04-01 14:55:51,857 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:51,857 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:55:51,858 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48772 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:55:51,858 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/C is initiating minor compaction (all files) 2023-04-01 14:55:51,859 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/C in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:51,859 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/cfdafa2650484cdebd8bbae2c84a9a48, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/86b4bd188a7f40f6b30691ed3d1fad8a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/18fb72ac835644aebf0e158434eeeb59, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/23fbfc9d9acf4bf3a78b8da7f1a39533] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=47.6 K 2023-04-01 14:55:51,859 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting cfdafa2650484cdebd8bbae2c84a9a48, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1680360949263 2023-04-01 14:55:51,860 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 86b4bd188a7f40f6b30691ed3d1fad8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680360949329 2023-04-01 14:55:51,860 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 18fb72ac835644aebf0e158434eeeb59, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1680360949967 2023-04-01 14:55:51,860 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 23fbfc9d9acf4bf3a78b8da7f1a39533, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680360950636 2023-04-01 14:55:51,867 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#C#compaction#455 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:52,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:52,119 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:52,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48116 deadline: 1680361012135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361012136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48086 deadline: 1680361012137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361012140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,238 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/2558606886204e4187525ab6cd6d3698 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2558606886204e4187525ab6cd6d3698 2023-04-01 14:55:52,243 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into 2558606886204e4187525ab6cd6d3698(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:52,243 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:52,243 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=12, startTime=1680360951807; duration=0sec 2023-04-01 14:55:52,244 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:52,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361012241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361012243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361012244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361012244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,263 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/34f66e69bfd54f2985477649529566b0 2023-04-01 14:55:52,267 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/044d8ec440d542c9a380c5dba51a99cb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/044d8ec440d542c9a380c5dba51a99cb 2023-04-01 14:55:52,271 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/044d8ec440d542c9a380c5dba51a99cb, entries=150, sequenceid=281, filesize=11.9 K 2023-04-01 14:55:52,272 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/7b5d598e67b64916a4dcc2bf36e1fca4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7b5d598e67b64916a4dcc2bf36e1fca4 2023-04-01 14:55:52,276 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7b5d598e67b64916a4dcc2bf36e1fca4, entries=150, sequenceid=281, filesize=11.9 K 2023-04-01 14:55:52,281 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/34f66e69bfd54f2985477649529566b0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/34f66e69bfd54f2985477649529566b0 2023-04-01 14:55:52,282 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/f346e12e5fa84d31a1939a06eac7053f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f346e12e5fa84d31a1939a06eac7053f 2023-04-01 14:55:52,286 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/34f66e69bfd54f2985477649529566b0, entries=150, sequenceid=281, filesize=11.9 K 2023-04-01 14:55:52,287 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 477ms, sequenceid=281, compaction requested=false 2023-04-01 14:55:52,287 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/C of 2ddd6fcff99c4aa857940e9dc8e40f15 into f346e12e5fa84d31a1939a06eac7053f(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:52,287 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:52,287 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:52,287 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:52,287 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/C, priority=12, startTime=1680360951807; duration=0sec 2023-04-01 14:55:52,287 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,288 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:52,289 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:52,290 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,290 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:52,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:52,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791edcc1[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:52,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:52,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,294 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:52,294 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,294 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:52,295 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:52,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:52,295 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:52,296 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:52,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:52,300 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:52,300 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:52,300 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:52,300 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:52,300 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:52,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:52,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:52,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,308 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:52,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:52,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:52,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:52,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:52,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:52,311 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:52,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:52,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:52,312 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,312 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,312 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,312 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,312 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:52,312 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:52,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:52,412 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:52,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:52,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:52,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:52,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:52,415 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:52,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:52,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:52,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:52,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:52,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:52,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:52,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:52,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:52,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:52,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:52,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:52,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:52,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:52,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:52,420 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:52,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:52,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:52,420 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:55:52,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:52,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:52,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:52,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:52,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:52,420 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:52,432 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/edbc0a52a2314d0782c5dfe5e20f514b 2023-04-01 14:55:52,444 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/cd4b0c019fa34a0dbcdba2218486d0b3 2023-04-01 14:55:52,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:52,449 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:52,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361012459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361012464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361012465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361012465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361012566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361012568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361012571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361012574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48080 deadline: 1680361012743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361012770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361012770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361012776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:52,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361012779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,861 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/8068d87861f64b7a8abf9f3af605e874 2023-04-01 14:55:52,865 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/edbc0a52a2314d0782c5dfe5e20f514b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/edbc0a52a2314d0782c5dfe5e20f514b 2023-04-01 14:55:52,869 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/edbc0a52a2314d0782c5dfe5e20f514b, entries=150, sequenceid=303, filesize=11.9 K 2023-04-01 14:55:52,870 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/cd4b0c019fa34a0dbcdba2218486d0b3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/cd4b0c019fa34a0dbcdba2218486d0b3 2023-04-01 14:55:52,874 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/cd4b0c019fa34a0dbcdba2218486d0b3, entries=150, sequenceid=303, filesize=11.9 K 2023-04-01 14:55:52,874 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/8068d87861f64b7a8abf9f3af605e874 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8068d87861f64b7a8abf9f3af605e874 2023-04-01 14:55:52,890 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8068d87861f64b7a8abf9f3af605e874, entries=150, sequenceid=303, filesize=11.9 K 2023-04-01 14:55:52,891 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 471ms, sequenceid=303, compaction requested=true 2023-04-01 14:55:52,891 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:52,891 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:52,891 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:52,891 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:52,891 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,892 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:52,892 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:52,893 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,893 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@282502a0[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:52,895 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:52,895 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,899 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:52,899 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,899 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:52,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:52,908 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:52,908 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:52,909 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:52,910 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:52,910 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:52,910 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:52,910 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:52,910 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:52,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:52,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:52,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:52,914 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,915 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,919 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:52,919 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:52,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:52,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:52,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:52,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:52,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:52,920 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:52,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:52,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:52,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:52,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:52,921 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:52,921 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:52,921 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:53,021 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:53,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:53,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:53,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:53,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:53,024 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:53,024 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:53,024 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:53,024 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:53,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:53,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:53,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:53,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:53,026 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:53,027 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,027 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:53,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:53,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:53,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:53,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:53,029 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:53,030 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:53,030 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:53,030 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:53,030 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:53,030 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:53,030 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:53,030 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:53,030 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:53,037 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/c1d09a0d98c44ed6a201ac86819a7c47 2023-04-01 14:55:53,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:53,079 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:53,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361013104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361013104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361013105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361013106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361013212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361013212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361013214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361013215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361013419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361013420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361013422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361013423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,452 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/e9bd7cf001d2464fa90b71e7a16ea632 2023-04-01 14:55:53,463 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/0ac8c71699d24b378347db046d1f2905 2023-04-01 14:55:53,468 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/c1d09a0d98c44ed6a201ac86819a7c47 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c1d09a0d98c44ed6a201ac86819a7c47 2023-04-01 14:55:53,471 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c1d09a0d98c44ed6a201ac86819a7c47, entries=150, sequenceid=317, filesize=11.9 K 2023-04-01 14:55:53,472 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/e9bd7cf001d2464fa90b71e7a16ea632 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e9bd7cf001d2464fa90b71e7a16ea632 2023-04-01 14:55:53,475 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e9bd7cf001d2464fa90b71e7a16ea632, entries=150, sequenceid=317, filesize=11.9 K 2023-04-01 14:55:53,476 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/0ac8c71699d24b378347db046d1f2905 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/0ac8c71699d24b378347db046d1f2905 2023-04-01 14:55:53,479 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/0ac8c71699d24b378347db046d1f2905, entries=150, sequenceid=317, filesize=11.9 K 2023-04-01 14:55:53,480 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 450ms, sequenceid=317, compaction requested=true 2023-04-01 14:55:53,480 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:53,480 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:53,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:53,483 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:53,483 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:53,483 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:53,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53fd0ce5[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:53,485 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:53,485 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:53,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:53,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:53,486 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:53,486 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:53,486 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:53,486 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,486 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:53,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:53,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:53,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:53,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:53,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:53,493 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:53,493 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:53,493 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:53,493 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:53,493 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:53,495 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:53,495 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:53,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:53,495 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:53,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:53,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:53,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:53,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:53,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:53,499 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:53,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,502 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:53,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:53,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:53,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:53,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:53,502 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:53,502 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:53,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:53,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:53,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:53,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:53,502 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:53,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:53,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:53,602 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:53,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:53,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:53,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:53,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:53,606 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:53,606 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:53,606 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:53,606 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:53,606 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:53,606 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:53,608 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:53,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:53,608 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,608 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:53,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:53,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:53,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:53,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:53,617 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:53,617 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:53,617 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:55:53,618 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:53,618 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:53,618 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:53,618 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:53,618 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:53,618 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:53,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:53,729 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:53,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361013747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361013748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361013748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361013749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361013850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361013851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361013856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:53,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:53,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361013856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,044 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/18a6975879fb41659cacedde8fa6d674 2023-04-01 14:55:54,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361014060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361014061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361014061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361014062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361014372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361014373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361014373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361014373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,455 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/8c5b94ee725e4f57a1dc6aa68e6afbb1 2023-04-01 14:55:54,477 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/f3b19c7e7e544b3591216c58f8cdfc63 2023-04-01 14:55:54,481 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/18a6975879fb41659cacedde8fa6d674 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/18a6975879fb41659cacedde8fa6d674 2023-04-01 14:55:54,486 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/18a6975879fb41659cacedde8fa6d674, entries=150, sequenceid=335, filesize=11.9 K 2023-04-01 14:55:54,486 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/8c5b94ee725e4f57a1dc6aa68e6afbb1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8c5b94ee725e4f57a1dc6aa68e6afbb1 2023-04-01 14:55:54,490 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8c5b94ee725e4f57a1dc6aa68e6afbb1, entries=150, sequenceid=335, filesize=11.9 K 2023-04-01 14:55:54,490 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/f3b19c7e7e544b3591216c58f8cdfc63 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f3b19c7e7e544b3591216c58f8cdfc63 2023-04-01 14:55:54,494 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f3b19c7e7e544b3591216c58f8cdfc63, entries=150, sequenceid=335, filesize=11.9 K 2023-04-01 14:55:54,495 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 878ms, sequenceid=335, compaction requested=true 2023-04-01 14:55:54,495 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:54,495 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:54,498 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:54,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d0c9377[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:54,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:54,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,501 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:54,502 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,502 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:54,503 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:54,503 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:54,503 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:54,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:54,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:54,506 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:54,506 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:54,507 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:54,507 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:54,507 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:54,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:54,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:54,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,509 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,511 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:54,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:54,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:54,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:54,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:54,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:54,516 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:54,516 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:54,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:54,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:54,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,516 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:54,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,616 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:54,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:54,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:54,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:54,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:54,656 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:54,656 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:54,656 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:54,656 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:54,656 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,656 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:54,657 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:54,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:54,658 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,658 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,658 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:54,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:54,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:54,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:54,672 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:54,672 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:54,672 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:54,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:54,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:54,692 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:54,692 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:54,692 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:54,692 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:54,692 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:54,692 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:54,693 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:54,693 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:54,693 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:54,726 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/a32203114bd2459e84f5395741bdfe49 2023-04-01 14:55:54,748 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/6e047409777b47738c28462cc0ca8061 2023-04-01 14:55:54,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:54,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:54,829 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/bf6dbcf2668a4fc7988be93b606a9e25 2023-04-01 14:55:54,835 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/a32203114bd2459e84f5395741bdfe49 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/a32203114bd2459e84f5395741bdfe49 2023-04-01 14:55:54,839 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/a32203114bd2459e84f5395741bdfe49, entries=150, sequenceid=349, filesize=11.9 K 2023-04-01 14:55:54,839 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/6e047409777b47738c28462cc0ca8061 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/6e047409777b47738c28462cc0ca8061 2023-04-01 14:55:54,852 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/6e047409777b47738c28462cc0ca8061, entries=150, sequenceid=349, filesize=11.9 K 2023-04-01 14:55:54,854 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/bf6dbcf2668a4fc7988be93b606a9e25 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf6dbcf2668a4fc7988be93b606a9e25 2023-04-01 14:55:54,872 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf6dbcf2668a4fc7988be93b606a9e25, entries=150, sequenceid=349, filesize=11.9 K 2023-04-01 14:55:54,873 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=60.38 KB/61830 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 181ms, sequenceid=349, compaction requested=true 2023-04-01 14:55:54,873 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:54,873 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:54,873 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:54,873 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:54,873 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:54,877 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,879 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62fbb2c3[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:54,879 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:54,879 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,882 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:54,882 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,882 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:54,884 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:54,884 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:54,884 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:54,884 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:54,887 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 14:55:54,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:54,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:54,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:54,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:54,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:54,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:54,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:54,889 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:54,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:54,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:54,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:54,889 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:54,895 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:54,895 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,895 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:54,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:54,895 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:54,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:54,899 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:54,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:54,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:54,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:54,901 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:54,902 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:54,902 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:54,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:54,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:54,902 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:54,902 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:54,902 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,902 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:54,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:54,902 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:54,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:54,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/d55066b5e4534ca8a1d5468e4083411d 2023-04-01 14:55:54,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361014936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361014944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/a5df6c2338fc4f68b2021ac44e24fed5 2023-04-01 14:55:54,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361014946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361014946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:54,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361014953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:54,978 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/fb2b3379a55a44d598c647ae12f12492 2023-04-01 14:55:54,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/d55066b5e4534ca8a1d5468e4083411d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d55066b5e4534ca8a1d5468e4083411d 2023-04-01 14:55:54,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d55066b5e4534ca8a1d5468e4083411d, entries=200, sequenceid=362, filesize=14.3 K 2023-04-01 14:55:54,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/a5df6c2338fc4f68b2021ac44e24fed5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a5df6c2338fc4f68b2021ac44e24fed5 2023-04-01 14:55:55,000 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a5df6c2338fc4f68b2021ac44e24fed5, entries=150, sequenceid=362, filesize=11.9 K 2023-04-01 14:55:55,001 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/fb2b3379a55a44d598c647ae12f12492 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/fb2b3379a55a44d598c647ae12f12492 2023-04-01 14:55:55,002 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:55,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/fb2b3379a55a44d598c647ae12f12492, entries=150, sequenceid=362, filesize=11.9 K 2023-04-01 14:55:55,005 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 118ms, sequenceid=362, compaction requested=true 2023-04-01 14:55:55,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:55,005 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:55,005 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-01 14:55:55,005 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-01 14:55:55,005 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:55,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:55,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:55,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:55,016 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85916 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-01 14:55:55,016 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:55:55,017 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,017 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2558606886204e4187525ab6cd6d3698, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7b5d598e67b64916a4dcc2bf36e1fca4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/cd4b0c019fa34a0dbcdba2218486d0b3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e9bd7cf001d2464fa90b71e7a16ea632, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8c5b94ee725e4f57a1dc6aa68e6afbb1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/6e047409777b47738c28462cc0ca8061, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a5df6c2338fc4f68b2021ac44e24fed5] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=83.9 K 2023-04-01 14:55:55,017 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 88356 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-01 14:55:55,017 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/A is initiating minor compaction (all files) 2023-04-01 14:55:55,017 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/A in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,017 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/0d4d8ad2a2fb4426814e51b4faaf069c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/044d8ec440d542c9a380c5dba51a99cb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/edbc0a52a2314d0782c5dfe5e20f514b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c1d09a0d98c44ed6a201ac86819a7c47, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/18a6975879fb41659cacedde8fa6d674, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/a32203114bd2459e84f5395741bdfe49, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d55066b5e4534ca8a1d5468e4083411d] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=86.3 K 2023-04-01 14:55:55,018 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2558606886204e4187525ab6cd6d3698, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680360950636 2023-04-01 14:55:55,018 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 0d4d8ad2a2fb4426814e51b4faaf069c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680360950636 2023-04-01 14:55:55,018 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 7b5d598e67b64916a4dcc2bf36e1fca4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680360950978 2023-04-01 14:55:55,019 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 044d8ec440d542c9a380c5dba51a99cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680360950978 2023-04-01 14:55:55,019 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cd4b0c019fa34a0dbcdba2218486d0b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680360952134 2023-04-01 14:55:55,019 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting edbc0a52a2314d0782c5dfe5e20f514b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680360952134 2023-04-01 14:55:55,020 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e9bd7cf001d2464fa90b71e7a16ea632, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680360952463 2023-04-01 14:55:55,020 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting c1d09a0d98c44ed6a201ac86819a7c47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680360952463 2023-04-01 14:55:55,020 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 8c5b94ee725e4f57a1dc6aa68e6afbb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680360953099 2023-04-01 14:55:55,020 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 18a6975879fb41659cacedde8fa6d674, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680360953099 2023-04-01 14:55:55,021 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6e047409777b47738c28462cc0ca8061, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680360953746 2023-04-01 14:55:55,021 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting a32203114bd2459e84f5395741bdfe49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680360953746 2023-04-01 14:55:55,021 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a5df6c2338fc4f68b2021ac44e24fed5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680360954839 2023-04-01 14:55:55,022 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d55066b5e4534ca8a1d5468e4083411d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680360954839 2023-04-01 14:55:55,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:55,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:55,024 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:55,024 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:55,024 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:55,024 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:55,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,027 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:55,027 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,028 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,028 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:55,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:55,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:55,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:55,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:55,040 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:55,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:55,044 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:55,044 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,044 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:55,044 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:55:55,045 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:55,045 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:55,045 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:55,045 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:55,045 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:55,045 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:55,049 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#A#compaction#471 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:55,056 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#472 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:55,066 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:55,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:55,095 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/65d59e3faff14edab5c9a2c901f8a3f7 2023-04-01 14:55:55,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361015089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361015090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361015091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361015097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361015103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,136 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/8b26b40b60804873bd35314fa705f193 2023-04-01 14:55:55,158 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/ea30ee8fb47148b381f66bb4b93758bb 2023-04-01 14:55:55,164 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/65d59e3faff14edab5c9a2c901f8a3f7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/65d59e3faff14edab5c9a2c901f8a3f7 2023-04-01 14:55:55,168 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/65d59e3faff14edab5c9a2c901f8a3f7, entries=150, sequenceid=381, filesize=11.9 K 2023-04-01 14:55:55,168 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/8b26b40b60804873bd35314fa705f193 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8b26b40b60804873bd35314fa705f193 2023-04-01 14:55:55,175 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8b26b40b60804873bd35314fa705f193, entries=150, sequenceid=381, filesize=11.9 K 2023-04-01 14:55:55,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/ea30ee8fb47148b381f66bb4b93758bb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ea30ee8fb47148b381f66bb4b93758bb 2023-04-01 14:55:55,185 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ea30ee8fb47148b381f66bb4b93758bb, entries=150, sequenceid=381, filesize=11.9 K 2023-04-01 14:55:55,185 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 141ms, sequenceid=381, compaction requested=true 2023-04-01 14:55:55,185 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:55,186 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,186 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:55,186 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:55,186 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,186 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:55,186 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:55,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@624b8c3e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:55,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:55,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,192 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:55,192 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:55,193 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:55,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:55,194 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:55,194 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:55,194 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:55,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:55,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:55,197 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:55,197 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:55,197 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:55,197 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:55,197 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:55,199 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,199 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:55,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,199 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,201 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,205 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,205 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,205 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,205 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,205 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,205 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,205 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,206 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:55,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:55,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:55,206 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:55,206 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:55,206 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:55,206 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:55,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:55,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:55,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:55,222 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:55,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:55,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:55,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:55,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:55,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:55,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:55,262 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/68666cc5363b412ea262cb48487bb737 2023-04-01 14:55:55,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361015261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361015262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361015264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361015265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361015278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,306 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:55,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:55,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:55,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:55,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:55,342 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:55,342 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:55,342 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:55,342 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:55,343 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,343 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,344 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:55,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:55,345 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,345 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:55,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:55,348 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:55,348 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:55,348 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:55,363 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:55,363 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:55,364 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,364 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:55,364 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:55,364 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:55,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:55,365 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,366 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,366 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@596a2eb6[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:55,368 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:55,368 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,371 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:55,372 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,372 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:55,374 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:55,374 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,374 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:55,374 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:55,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:55,377 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:55,377 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:55,377 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:55,377 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:55,377 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:55,381 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,381 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:55,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:55,381 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,383 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,384 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,388 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:55,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:55,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:55,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,389 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,389 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:55,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:55,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:55,389 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:55,389 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:55,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:55,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361015380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361015380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361015381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361015381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361015394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,490 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:55,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:55,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:55,496 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/cc057543248c4f2e95b5055a140210d0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/cc057543248c4f2e95b5055a140210d0 2023-04-01 14:55:55,501 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/A of 2ddd6fcff99c4aa857940e9dc8e40f15 into cc057543248c4f2e95b5055a140210d0(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:55,501 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:55,501 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/A, priority=9, startTime=1680360955005; duration=0sec 2023-04-01 14:55:55,502 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:55,502 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-01 14:55:55,504 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98102 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-01 14:55:55,505 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/C is initiating minor compaction (all files) 2023-04-01 14:55:55,505 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/C in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,505 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f346e12e5fa84d31a1939a06eac7053f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/34f66e69bfd54f2985477649529566b0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8068d87861f64b7a8abf9f3af605e874, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/0ac8c71699d24b378347db046d1f2905, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f3b19c7e7e544b3591216c58f8cdfc63, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf6dbcf2668a4fc7988be93b606a9e25, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/fb2b3379a55a44d598c647ae12f12492, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ea30ee8fb47148b381f66bb4b93758bb] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=95.8 K 2023-04-01 14:55:55,505 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting f346e12e5fa84d31a1939a06eac7053f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680360950636 2023-04-01 14:55:55,505 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 34f66e69bfd54f2985477649529566b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680360950978 2023-04-01 14:55:55,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:55,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:55,508 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:55,508 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:55,508 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 8068d87861f64b7a8abf9f3af605e874, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680360952134 2023-04-01 14:55:55,508 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:55,508 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:55,508 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 0ac8c71699d24b378347db046d1f2905, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680360952463 2023-04-01 14:55:55,509 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,509 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,509 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting f3b19c7e7e544b3591216c58f8cdfc63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680360953099 2023-04-01 14:55:55,509 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting bf6dbcf2668a4fc7988be93b606a9e25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680360953746 2023-04-01 14:55:55,510 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting fb2b3379a55a44d598c647ae12f12492, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680360954839 2023-04-01 14:55:55,510 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting ea30ee8fb47148b381f66bb4b93758bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680360954942 2023-04-01 14:55:55,511 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:55,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:55,511 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,511 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,511 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:55,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:55,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:55,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:55,532 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:55,532 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:55,532 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,533 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:55,535 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:55,535 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,535 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:55,535 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,537 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/357a256b4a3d43bca28f2688f1c0e8d7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/357a256b4a3d43bca28f2688f1c0e8d7 2023-04-01 14:55:55,541 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into 357a256b4a3d43bca28f2688f1c0e8d7(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:55,541 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:55,541 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=9, startTime=1680360955005; duration=0sec 2023-04-01 14:55:55,541 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:55,546 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#C#compaction#478 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:55,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f499c7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:55,556 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:55,556 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,558 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,558 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:55,558 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:55,558 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:55,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,559 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:55,560 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:55,563 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:55,563 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:55,563 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:55,563 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:55,563 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:55,566 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,567 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,567 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,568 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,569 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:55,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:55,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:55,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,574 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:55,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:55,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,575 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,575 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:55,575 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:55,575 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:55,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:55,581 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/da73d16c721c475fb1d4d9a6b474ea92 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/da73d16c721c475fb1d4d9a6b474ea92 2023-04-01 14:55:55,586 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/C of 2ddd6fcff99c4aa857940e9dc8e40f15 into da73d16c721c475fb1d4d9a6b474ea92(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:55,586 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:55,586 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/C, priority=8, startTime=1680360955006; duration=0sec 2023-04-01 14:55:55,586 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:55,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361015597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361015598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361015598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361015598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361015604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,675 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:55,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:55,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:55,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:55,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:55,679 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:55,679 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:55,679 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:55,679 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:55,679 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,679 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,680 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:55,680 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,683 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:55,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:55,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:55,692 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,692 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:55,692 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:55,692 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:55,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:55,695 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:55,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f2654b0[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:55,697 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:55,697 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,698 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/e22514a0910b4b7da5b1b918892855d6 2023-04-01 14:55:55,698 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,699 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,699 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:55,702 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:55,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:55,702 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:55,708 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:55,708 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:55,708 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:55,708 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:55,708 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:55,710 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,710 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:55,710 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,711 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,713 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:55,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:55,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:55,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:55,720 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,720 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,720 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,720 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,720 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:55,720 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,720 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:55,720 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:55,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:55,720 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:55,720 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:55,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/b617ac809a874305b218d917726fe730 2023-04-01 14:55:55,745 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/68666cc5363b412ea262cb48487bb737 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/68666cc5363b412ea262cb48487bb737 2023-04-01 14:55:55,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/68666cc5363b412ea262cb48487bb737, entries=200, sequenceid=395, filesize=14.3 K 2023-04-01 14:55:55,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/e22514a0910b4b7da5b1b918892855d6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e22514a0910b4b7da5b1b918892855d6 2023-04-01 14:55:55,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e22514a0910b4b7da5b1b918892855d6, entries=150, sequenceid=395, filesize=11.9 K 2023-04-01 14:55:55,781 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/b617ac809a874305b218d917726fe730 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b617ac809a874305b218d917726fe730 2023-04-01 14:55:55,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b617ac809a874305b218d917726fe730, entries=150, sequenceid=395, filesize=11.9 K 2023-04-01 14:55:55,786 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 564ms, sequenceid=395, compaction requested=true 2023-04-01 14:55:55,786 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:55,786 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:55,787 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:55,787 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:55,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:55,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:55,788 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39850 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:55,788 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/A is initiating minor compaction (all files) 2023-04-01 14:55:55,788 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:55,788 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/A in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,788 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:55:55,788 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/cc057543248c4f2e95b5055a140210d0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/65d59e3faff14edab5c9a2c901f8a3f7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/68666cc5363b412ea262cb48487bb737] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=38.9 K 2023-04-01 14:55:55,788 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,788 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/357a256b4a3d43bca28f2688f1c0e8d7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8b26b40b60804873bd35314fa705f193, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e22514a0910b4b7da5b1b918892855d6] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=36.5 K 2023-04-01 14:55:55,788 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting cc057543248c4f2e95b5055a140210d0, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680360954839 2023-04-01 14:55:55,792 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 357a256b4a3d43bca28f2688f1c0e8d7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680360954839 2023-04-01 14:55:55,792 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 65d59e3faff14edab5c9a2c901f8a3f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680360954942 2023-04-01 14:55:55,793 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 8b26b40b60804873bd35314fa705f193, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680360954942 2023-04-01 14:55:55,793 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e22514a0910b4b7da5b1b918892855d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680360955094 2023-04-01 14:55:55,793 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 68666cc5363b412ea262cb48487bb737, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680360955090 2023-04-01 14:55:55,820 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:55,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:55,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:55,827 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#480 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:55,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:55,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:55,832 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:55,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:55,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:55,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:55,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,832 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,835 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:55,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:55,835 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,837 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,837 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:55,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:55,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:55,841 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#A#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:55,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:55,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:55,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:55,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:55,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:55,884 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:55,896 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:55,896 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:55,896 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:55:55,896 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:55,896 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:55,896 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:55,897 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:55,897 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:55,897 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:55,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:55,914 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:55,939 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/00edfd55ca76480a99658f9d320ab91a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/00edfd55ca76480a99658f9d320ab91a 2023-04-01 14:55:55,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361015927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361015929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361015930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361015935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:55,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361015936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:55,945 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/A of 2ddd6fcff99c4aa857940e9dc8e40f15 into 00edfd55ca76480a99658f9d320ab91a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:55,945 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:55,945 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/A, priority=13, startTime=1680360955786; duration=0sec 2023-04-01 14:55:55,945 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:55,945 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-01 14:55:55,946 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-01 14:55:55,946 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-01 14:55:55,947 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. because compaction request was cancelled 2023-04-01 14:55:55,958 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/693213ca2376448081d7ef7cb2b8e0ed as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/693213ca2376448081d7ef7cb2b8e0ed 2023-04-01 14:55:55,969 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into 693213ca2376448081d7ef7cb2b8e0ed(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:55,970 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:55,970 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=13, startTime=1680360955787; duration=0sec 2023-04-01 14:55:55,970 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:55,970 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/35e38d944ffd4095b108b865c0113a1a 2023-04-01 14:55:55,996 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/5d703202d5494d2fb34488c72610bac3 2023-04-01 14:55:56,024 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/da4c4ec06f664f0cbec5a729f6e39c10 2023-04-01 14:55:56,028 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/35e38d944ffd4095b108b865c0113a1a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/35e38d944ffd4095b108b865c0113a1a 2023-04-01 14:55:56,033 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/35e38d944ffd4095b108b865c0113a1a, entries=150, sequenceid=416, filesize=11.9 K 2023-04-01 14:55:56,034 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/5d703202d5494d2fb34488c72610bac3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5d703202d5494d2fb34488c72610bac3 2023-04-01 14:55:56,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361016043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361016043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361016043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361016043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361016043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,058 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5d703202d5494d2fb34488c72610bac3, entries=150, sequenceid=416, filesize=11.9 K 2023-04-01 14:55:56,060 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/da4c4ec06f664f0cbec5a729f6e39c10 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/da4c4ec06f664f0cbec5a729f6e39c10 2023-04-01 14:55:56,063 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/da4c4ec06f664f0cbec5a729f6e39c10, entries=150, sequenceid=416, filesize=11.9 K 2023-04-01 14:55:56,064 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 168ms, sequenceid=416, compaction requested=true 2023-04-01 14:55:56,064 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:56,064 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:56,064 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:56,065 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:56,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:56,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,067 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51b21f16[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:56,069 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:56,069 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,071 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,071 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:56,072 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:56,072 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:56,072 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,072 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:56,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:56,075 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:56,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:56,078 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:56,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:56,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:56,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:56,078 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:56,080 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,080 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:56,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,080 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:56,087 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,089 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,091 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:56,091 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:56,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:56,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:56,092 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,092 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:56,092 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:56,092 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,092 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,092 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,092 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,092 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:56,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:56,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:56,093 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:56,093 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:56,194 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:56,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:56,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:56,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:56,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:56,200 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:56,200 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:56,200 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:56,200 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:56,200 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,201 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:56,202 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:56,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:56,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:56,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,204 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,204 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:56,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:56,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:56,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:56,220 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:56,220 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:56,221 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:56,221 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:56,221 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:56,221 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:56,221 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:56,221 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:56,221 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:56,221 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:56,221 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:56,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:56,256 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:56,257 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/55220ae246934adbae18621d67723774 2023-04-01 14:55:56,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361016277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361016281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361016287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361016288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361016288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361016389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361016390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361016393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361016396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361016397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361016597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361016597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361016600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361016605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361016605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,674 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/7a8cd61b26fe4aaf8f6f1a228e9d0209 2023-04-01 14:55:56,687 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/ece89fb609e14a27bafc0ec997dee17a 2023-04-01 14:55:56,691 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/55220ae246934adbae18621d67723774 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/55220ae246934adbae18621d67723774 2023-04-01 14:55:56,695 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/55220ae246934adbae18621d67723774, entries=150, sequenceid=432, filesize=11.9 K 2023-04-01 14:55:56,696 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/7a8cd61b26fe4aaf8f6f1a228e9d0209 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7a8cd61b26fe4aaf8f6f1a228e9d0209 2023-04-01 14:55:56,699 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7a8cd61b26fe4aaf8f6f1a228e9d0209, entries=150, sequenceid=432, filesize=11.9 K 2023-04-01 14:55:56,700 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/ece89fb609e14a27bafc0ec997dee17a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ece89fb609e14a27bafc0ec997dee17a 2023-04-01 14:55:56,703 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ece89fb609e14a27bafc0ec997dee17a, entries=150, sequenceid=432, filesize=11.9 K 2023-04-01 14:55:56,704 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 483ms, sequenceid=432, compaction requested=true 2023-04-01 14:55:56,704 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:56,704 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:56,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:56,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:56,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:56,704 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:56,709 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,709 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59a59408[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:56,710 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:56,710 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,716 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,717 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:56,717 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:56,717 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:56,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,718 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:56,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:56,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:56,719 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:56,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:56,722 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:56,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:56,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:56,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:56,722 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:56,725 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,725 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,725 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:56,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,726 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,727 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:56,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:56,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:56,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:56,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:56,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:56,738 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:56,738 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:56,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,739 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:56,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:56,739 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:56,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:56,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:56,838 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:56,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:56,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:56,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:56,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:56,845 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:56,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:56,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:56,845 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:56,846 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,846 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:56,847 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:56,847 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:56,847 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,847 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:56,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:56,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:56,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:56,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:56,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:56,849 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:56,849 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:55:56,851 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:56,853 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:56,853 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:56,853 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:56,853 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:56,853 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:56,881 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/c8a33789e19b4438907fffef3cd5c9d1 2023-04-01 14:55:56,897 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/b717596377c34b56802c0be06272cfc2 2023-04-01 14:55:56,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:56,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:56,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361016923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361016924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361016924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361016927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:56,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:56,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361016932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361017034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361017038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361017038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361017038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361017038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361017244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361017244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361017245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361017246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361017246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,322 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/a995aab5493e459b8ac18c9b5b2bd6af 2023-04-01 14:55:57,326 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/c8a33789e19b4438907fffef3cd5c9d1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c8a33789e19b4438907fffef3cd5c9d1 2023-04-01 14:55:57,330 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c8a33789e19b4438907fffef3cd5c9d1, entries=150, sequenceid=451, filesize=11.9 K 2023-04-01 14:55:57,331 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/b717596377c34b56802c0be06272cfc2 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b717596377c34b56802c0be06272cfc2 2023-04-01 14:55:57,335 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b717596377c34b56802c0be06272cfc2, entries=150, sequenceid=451, filesize=11.9 K 2023-04-01 14:55:57,336 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/a995aab5493e459b8ac18c9b5b2bd6af as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/a995aab5493e459b8ac18c9b5b2bd6af 2023-04-01 14:55:57,342 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/a995aab5493e459b8ac18c9b5b2bd6af, entries=150, sequenceid=451, filesize=11.9 K 2023-04-01 14:55:57,343 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 494ms, sequenceid=451, compaction requested=true 2023-04-01 14:55:57,343 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:57,343 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:57,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,345 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:57,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d6b045a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:57,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:57,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,348 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,348 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:57,348 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:57,348 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:57,348 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,349 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:57,351 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,352 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:57,352 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,352 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:57,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:57,353 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:57,354 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:57,354 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:57,354 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:57,354 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:57,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:57,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,356 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,357 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,359 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:57,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:57,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:57,359 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:57,359 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:57,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,360 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:57,361 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:57,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:57,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:57,460 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:57,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:57,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:57,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:57,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:57,463 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:57,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:57,463 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:57,464 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:57,464 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,464 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,465 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:57,465 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:57,466 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,466 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:57,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:57,467 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:57,467 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:57,467 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:57,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:57,468 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:57,468 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:57,468 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:57,468 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 14:55:57,468 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:57,469 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:57,469 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:57,469 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:57,469 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:57,469 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:57,497 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/781c9e92594c46a7a44e3c3dad80c50e 2023-04-01 14:55:57,531 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/e188b7c8e4314d46852f7eeae8e3b52b 2023-04-01 14:55:57,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:57,556 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:57,568 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=464 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/14ae1682c53146e69901bc03a9fdcec8 2023-04-01 14:55:57,571 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/781c9e92594c46a7a44e3c3dad80c50e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/781c9e92594c46a7a44e3c3dad80c50e 2023-04-01 14:55:57,575 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/781c9e92594c46a7a44e3c3dad80c50e, entries=150, sequenceid=464, filesize=11.9 K 2023-04-01 14:55:57,576 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/e188b7c8e4314d46852f7eeae8e3b52b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e188b7c8e4314d46852f7eeae8e3b52b 2023-04-01 14:55:57,581 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e188b7c8e4314d46852f7eeae8e3b52b, entries=150, sequenceid=464, filesize=11.9 K 2023-04-01 14:55:57,582 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/14ae1682c53146e69901bc03a9fdcec8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/14ae1682c53146e69901bc03a9fdcec8 2023-04-01 14:55:57,585 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/14ae1682c53146e69901bc03a9fdcec8, entries=150, sequenceid=464, filesize=11.9 K 2023-04-01 14:55:57,586 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=67.09 KB/68700 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 118ms, sequenceid=464, compaction requested=true 2023-04-01 14:55:57,586 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:57,586 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:57,586 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:57,586 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:57,586 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:57,586 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:55:57,586 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:57,587 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:57,587 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:57,587 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:57,587 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:57,587 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:57,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:57,588 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:57,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:57,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70354693[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:57,594 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:57,594 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,595 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,595 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:57,595 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:57,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:57,595 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:57,597 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:57,597 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:57,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:57,600 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:57,600 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:57,600 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:57,600 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:57,600 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:57,602 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,602 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,603 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,604 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,605 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,609 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:57,609 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:57,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:57,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:57,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:57,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:57,609 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:57,609 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:57,610 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:57,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:57,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,610 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,622 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/f8a1b9f717ac40ad817807407a402b03 2023-04-01 14:55:57,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361017616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361017616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/85fcc3c43d1244aab8c2853e44acd15f 2023-04-01 14:55:57,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361017626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361017627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361017629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,650 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/9f0a20b02beb4adcbb160d0a9637ca8c 2023-04-01 14:55:57,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/f8a1b9f717ac40ad817807407a402b03 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/f8a1b9f717ac40ad817807407a402b03 2023-04-01 14:55:57,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/f8a1b9f717ac40ad817807407a402b03, entries=200, sequenceid=478, filesize=14.3 K 2023-04-01 14:55:57,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/85fcc3c43d1244aab8c2853e44acd15f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/85fcc3c43d1244aab8c2853e44acd15f 2023-04-01 14:55:57,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/85fcc3c43d1244aab8c2853e44acd15f, entries=150, sequenceid=478, filesize=11.9 K 2023-04-01 14:55:57,663 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/9f0a20b02beb4adcbb160d0a9637ca8c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/9f0a20b02beb4adcbb160d0a9637ca8c 2023-04-01 14:55:57,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/9f0a20b02beb4adcbb160d0a9637ca8c, entries=150, sequenceid=478, filesize=11.9 K 2023-04-01 14:55:57,667 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 81ms, sequenceid=478, compaction requested=true 2023-04-01 14:55:57,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:57,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:57,668 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 14:55:57,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:57,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 14:55:57,668 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-01 14:55:57,671 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76510 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 14:55:57,671 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/A is initiating minor compaction (all files) 2023-04-01 14:55:57,671 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/A in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:57,671 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/00edfd55ca76480a99658f9d320ab91a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/35e38d944ffd4095b108b865c0113a1a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/55220ae246934adbae18621d67723774, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c8a33789e19b4438907fffef3cd5c9d1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/781c9e92594c46a7a44e3c3dad80c50e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/f8a1b9f717ac40ad817807407a402b03] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=74.7 K 2023-04-01 14:55:57,672 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 00edfd55ca76480a99658f9d320ab91a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680360955094 2023-04-01 14:55:57,672 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86188 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-01 14:55:57,672 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/C is initiating minor compaction (all files) 2023-04-01 14:55:57,672 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/C in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:57,672 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/da73d16c721c475fb1d4d9a6b474ea92, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b617ac809a874305b218d917726fe730, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/da4c4ec06f664f0cbec5a729f6e39c10, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ece89fb609e14a27bafc0ec997dee17a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/a995aab5493e459b8ac18c9b5b2bd6af, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/14ae1682c53146e69901bc03a9fdcec8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/9f0a20b02beb4adcbb160d0a9637ca8c] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=84.2 K 2023-04-01 14:55:57,672 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 35e38d944ffd4095b108b865c0113a1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680360955262 2023-04-01 14:55:57,673 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting da73d16c721c475fb1d4d9a6b474ea92, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680360954942 2023-04-01 14:55:57,673 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 55220ae246934adbae18621d67723774, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680360955928 2023-04-01 14:55:57,673 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b617ac809a874305b218d917726fe730, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680360955094 2023-04-01 14:55:57,673 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting c8a33789e19b4438907fffef3cd5c9d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1680360956275 2023-04-01 14:55:57,684 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting da4c4ec06f664f0cbec5a729f6e39c10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680360955262 2023-04-01 14:55:57,684 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 781c9e92594c46a7a44e3c3dad80c50e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1680360956915 2023-04-01 14:55:57,685 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ece89fb609e14a27bafc0ec997dee17a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680360955928 2023-04-01 14:55:57,686 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting f8a1b9f717ac40ad817807407a402b03, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680360957559 2023-04-01 14:55:57,686 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a995aab5493e459b8ac18c9b5b2bd6af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1680360956275 2023-04-01 14:55:57,686 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 14ae1682c53146e69901bc03a9fdcec8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1680360956915 2023-04-01 14:55:57,687 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 9f0a20b02beb4adcbb160d0a9637ca8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680360957559 2023-04-01 14:55:57,710 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:57,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:57,710 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#A#compaction#497 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:57,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:57,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:57,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:57,713 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:57,713 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:57,713 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:57,713 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:57,714 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,714 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,715 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#C#compaction#498 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:57,717 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:57,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:57,722 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,723 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,723 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:57,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:57,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:57,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:57,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:57,723 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:57,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:57,741 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:55:57,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:57,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:57,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:57,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:57,741 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:57,741 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:57,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:57,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:57,745 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:57,745 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:57,746 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:57,746 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:57,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:57,750 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,750 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:57,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31bead2c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:57,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:57,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:57,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:57,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:57,754 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,754 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:57,756 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:57,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/5846d9a730ed4d5a916f1e8431b77c1d 2023-04-01 14:55:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:57,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:57,761 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:57,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:57,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:57,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:57,761 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:57,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:57,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:57,764 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:57,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:57,769 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:57,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:57,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:57,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:57,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,770 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,770 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,770 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:57,770 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:57,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:57,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361017766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361017767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361017767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361017768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361017778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,870 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:57,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:57,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:57,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:57,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:57,873 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:57,873 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:57,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:57,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:57,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,875 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:57,875 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:57,876 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,876 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:57,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:57,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:57,877 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:57,877 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:57,877 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:57,877 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:57,878 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:57,878 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:57,878 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,878 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:57,878 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:57,883 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,883 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361017879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,883 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,883 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44ce6ac7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:57,885 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:57,886 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361017879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361017879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361017879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,889 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:57,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:57,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:57,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:57,890 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,890 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:57,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:57,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:57,893 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:57,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361017888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,893 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:57,893 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:57,893 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:57,893 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:57,897 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,897 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:57,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,897 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:57,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:57,898 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,899 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:57,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:57,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:57,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:57,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:57,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:57,901 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:57,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:57,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:57,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,902 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:57,902 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:57,902 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:57,902 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,002 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:58,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:58,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:58,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:58,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:58,005 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:58,005 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:58,005 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:58,005 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:58,006 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,006 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,007 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:58,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,008 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:58,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:58,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:58,008 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,009 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:58,009 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:58,009 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:58,009 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:58,009 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:58,011 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:58,012 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,012 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,012 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:58,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:58,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3172efd0[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:58,014 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:58,014 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,022 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:58,022 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,022 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:58,023 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,023 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:58,024 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:58,024 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:58,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:58,026 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:58,026 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:58,026 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:58,026 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:58,026 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:58,031 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,031 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:58,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,031 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:58,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:58,032 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,033 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:58,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:58,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:58,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:58,035 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:58,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:58,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:58,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:58,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:58,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,036 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:58,036 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:58,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361018085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361018088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361018089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361018089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361018095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,136 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:58,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:58,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:58,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:58,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:58,143 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:58,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:58,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:58,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:58,144 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,144 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,144 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/421794020ccc48d3ad655d62305d1192 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/421794020ccc48d3ad655d62305d1192 2023-04-01 14:55:58,146 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:58,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:58,147 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,147 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:58,151 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/79b115cc0e7f4781a17c8ae9551c518d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/79b115cc0e7f4781a17c8ae9551c518d 2023-04-01 14:55:58,161 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/C of 2ddd6fcff99c4aa857940e9dc8e40f15 into 79b115cc0e7f4781a17c8ae9551c518d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:58,161 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:58,161 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/C, priority=9, startTime=1680360957668; duration=0sec 2023-04-01 14:55:58,161 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:58,161 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 14:55:58,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:58,162 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:58,163 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74070 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 14:55:58,163 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:55:58,163 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:58,163 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/693213ca2376448081d7ef7cb2b8e0ed, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5d703202d5494d2fb34488c72610bac3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7a8cd61b26fe4aaf8f6f1a228e9d0209, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b717596377c34b56802c0be06272cfc2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e188b7c8e4314d46852f7eeae8e3b52b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/85fcc3c43d1244aab8c2853e44acd15f] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=72.3 K 2023-04-01 14:55:58,163 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 693213ca2376448081d7ef7cb2b8e0ed, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680360955094 2023-04-01 14:55:58,164 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 5d703202d5494d2fb34488c72610bac3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1680360955262 2023-04-01 14:55:58,164 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 7a8cd61b26fe4aaf8f6f1a228e9d0209, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=432, earliestPutTs=1680360955928 2023-04-01 14:55:58,165 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b717596377c34b56802c0be06272cfc2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1680360956275 2023-04-01 14:55:58,165 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e188b7c8e4314d46852f7eeae8e3b52b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=464, earliestPutTs=1680360956915 2023-04-01 14:55:58,165 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 85fcc3c43d1244aab8c2853e44acd15f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680360957559 2023-04-01 14:55:58,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:58,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:58,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:58,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/99760ef96ea84bbcb9c7d41cdbe4042c 2023-04-01 14:55:58,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:58,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:58,176 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:58,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:58,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:58,176 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,177 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/A of 2ddd6fcff99c4aa857940e9dc8e40f15 into 421794020ccc48d3ad655d62305d1192(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:58,177 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:58,177 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/A, priority=10, startTime=1680360957667; duration=0sec 2023-04-01 14:55:58,178 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:58,181 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#501 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:58,182 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:58,182 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:58,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,184 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:58,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,184 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:58,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:58,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ba936b3[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:58,186 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:58,186 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,189 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:58,189 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:58,189 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:58,191 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:58,191 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:58,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,191 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:58,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:58,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:58,191 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:58,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:58,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:58,191 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:58,191 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:58,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:58,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:58,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:58,194 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,197 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:58,197 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,198 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:58,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:58,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:58,198 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,198 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:58,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:58,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:58,198 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:58,198 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:58,198 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,198 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:58,198 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,198 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,198 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,223 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/b6cebcf535b246808cb9e87c780e550a 2023-04-01 14:55:58,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/5846d9a730ed4d5a916f1e8431b77c1d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5846d9a730ed4d5a916f1e8431b77c1d 2023-04-01 14:55:58,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5846d9a730ed4d5a916f1e8431b77c1d, entries=200, sequenceid=497, filesize=14.3 K 2023-04-01 14:55:58,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/99760ef96ea84bbcb9c7d41cdbe4042c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/99760ef96ea84bbcb9c7d41cdbe4042c 2023-04-01 14:55:58,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/99760ef96ea84bbcb9c7d41cdbe4042c, entries=150, sequenceid=497, filesize=11.9 K 2023-04-01 14:55:58,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/b6cebcf535b246808cb9e87c780e550a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b6cebcf535b246808cb9e87c780e550a 2023-04-01 14:55:58,259 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b6cebcf535b246808cb9e87c780e550a, entries=150, sequenceid=497, filesize=11.9 K 2023-04-01 14:55:58,260 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 519ms, sequenceid=497, compaction requested=false 2023-04-01 14:55:58,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:58,298 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:58,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:58,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:58,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:58,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:58,302 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:58,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:58,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:58,302 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:58,304 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,304 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:58,305 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:58,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,306 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:58,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:58,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:58,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:58,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:58,312 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:58,320 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:58,320 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:58,320 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:58,320 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:58,320 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 14:55:58,320 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:58,321 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:58,321 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:58,321 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:58,321 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:58,321 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:58,345 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/6b95981e596643cbafc2e6e567435fd5 2023-04-01 14:55:58,372 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/2121bdafd88a4715b474562be5eee12a 2023-04-01 14:55:58,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:58,405 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:58,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361018426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361018426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361018427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361018432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361018435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361018536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361018536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361018537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361018545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361018545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,634 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/a71e0a7100ec4f5687d9529cc987530e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a71e0a7100ec4f5687d9529cc987530e 2023-04-01 14:55:58,638 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into a71e0a7100ec4f5687d9529cc987530e(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:58,638 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:58,638 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=10, startTime=1680360957668; duration=0sec 2023-04-01 14:55:58,638 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:58,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361018744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361018745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361018745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361018755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:58,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361018756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,796 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/7ab321cee76f469bab1907b60208900d 2023-04-01 14:55:58,800 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/6b95981e596643cbafc2e6e567435fd5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6b95981e596643cbafc2e6e567435fd5 2023-04-01 14:55:58,804 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6b95981e596643cbafc2e6e567435fd5, entries=150, sequenceid=512, filesize=11.9 K 2023-04-01 14:55:58,805 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/2121bdafd88a4715b474562be5eee12a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2121bdafd88a4715b474562be5eee12a 2023-04-01 14:55:58,810 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2121bdafd88a4715b474562be5eee12a, entries=150, sequenceid=512, filesize=11.9 K 2023-04-01 14:55:58,810 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/7ab321cee76f469bab1907b60208900d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7ab321cee76f469bab1907b60208900d 2023-04-01 14:55:58,814 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7ab321cee76f469bab1907b60208900d, entries=150, sequenceid=512, filesize=11.9 K 2023-04-01 14:55:58,827 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 507ms, sequenceid=512, compaction requested=true 2023-04-01 14:55:58,827 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:58,827 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:58,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,829 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:58,829 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,829 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:58,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:58,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3a3127[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:58,831 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:58,831 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:58,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,834 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:58,834 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:58,834 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:58,834 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:58,834 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:58,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:58,836 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:58,836 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:58,836 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:58,836 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:58,837 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:58,838 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:58,838 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,838 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:58,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,840 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,841 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,843 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,843 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,843 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,843 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,843 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,843 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:58,843 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,843 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:58,843 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:58,843 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:58,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:58,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:58,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:58,844 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:58,844 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:58,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:58,944 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:58,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:58,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:58,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:58,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:58,948 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:58,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:58,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:58,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:58,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,950 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:58,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:58,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:58,951 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:58,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:58,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:58,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:58,952 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:58,952 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:58,953 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:55:58,953 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:58,953 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:58,953 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:58,953 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:58,953 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:58,953 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:59,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:59,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:59,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361019067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361019069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361019104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361019104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361019104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361019205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361019205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361019216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361019216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361019216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,371 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/d0f0866690a940cbaf20f2b1d65a3732 2023-04-01 14:55:59,380 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/a94633b81cc649ed847372a0edb3cdc6 2023-04-01 14:55:59,390 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/05819d7750174df2ac2c9a3d25a12ab8 2023-04-01 14:55:59,398 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/d0f0866690a940cbaf20f2b1d65a3732 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d0f0866690a940cbaf20f2b1d65a3732 2023-04-01 14:55:59,401 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d0f0866690a940cbaf20f2b1d65a3732, entries=150, sequenceid=532, filesize=11.9 K 2023-04-01 14:55:59,401 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/a94633b81cc649ed847372a0edb3cdc6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a94633b81cc649ed847372a0edb3cdc6 2023-04-01 14:55:59,406 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a94633b81cc649ed847372a0edb3cdc6, entries=150, sequenceid=532, filesize=11.9 K 2023-04-01 14:55:59,407 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/05819d7750174df2ac2c9a3d25a12ab8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/05819d7750174df2ac2c9a3d25a12ab8 2023-04-01 14:55:59,410 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/05819d7750174df2ac2c9a3d25a12ab8, entries=150, sequenceid=532, filesize=11.9 K 2023-04-01 14:55:59,411 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 458ms, sequenceid=532, compaction requested=true 2023-04-01 14:55:59,411 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:59,411 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:59,412 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,412 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,413 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,413 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1815ee55[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:59,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:59,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:59,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:59,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:59,417 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 14:55:59,417 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:59,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:59,418 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:59,418 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:59,418 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:59,418 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:59,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:59,421 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:59,421 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:59,421 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:59,421 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:59,421 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:59,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,424 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:59,425 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,425 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,428 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,431 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:59,431 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:59,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:59,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:59,431 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,431 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,431 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:59,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:59,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:59,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:59,431 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,432 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,432 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:59,432 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,432 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,432 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,432 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:59,433 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/47eb033ec2b844b496ceeb692259984f 2023-04-01 14:55:59,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361019442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361019444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/a56a7367b25c4a8fad4a566ece309535 2023-04-01 14:55:59,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361019444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,463 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/f512032209a34a22b9e3a98e1682afd3 2023-04-01 14:55:59,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361019452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361019452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/47eb033ec2b844b496ceeb692259984f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/47eb033ec2b844b496ceeb692259984f 2023-04-01 14:55:59,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/47eb033ec2b844b496ceeb692259984f, entries=200, sequenceid=547, filesize=14.3 K 2023-04-01 14:55:59,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/a56a7367b25c4a8fad4a566ece309535 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a56a7367b25c4a8fad4a566ece309535 2023-04-01 14:55:59,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a56a7367b25c4a8fad4a566ece309535, entries=150, sequenceid=547, filesize=11.9 K 2023-04-01 14:55:59,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/f512032209a34a22b9e3a98e1682afd3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f512032209a34a22b9e3a98e1682afd3 2023-04-01 14:55:59,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f512032209a34a22b9e3a98e1682afd3, entries=150, sequenceid=547, filesize=11.9 K 2023-04-01 14:55:59,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 62ms, sequenceid=547, compaction requested=true 2023-04-01 14:55:59,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:59,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:55:59,479 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:55:59,479 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:55:59,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:59,480 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:55:59,481 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66968 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:55:59,481 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/A is initiating minor compaction (all files) 2023-04-01 14:55:59,481 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/A in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:59,481 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62088 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:55:59,481 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:55:59,481 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:59,481 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a71e0a7100ec4f5687d9529cc987530e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/99760ef96ea84bbcb9c7d41cdbe4042c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2121bdafd88a4715b474562be5eee12a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a94633b81cc649ed847372a0edb3cdc6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a56a7367b25c4a8fad4a566ece309535] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=60.6 K 2023-04-01 14:55:59,481 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/421794020ccc48d3ad655d62305d1192, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5846d9a730ed4d5a916f1e8431b77c1d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6b95981e596643cbafc2e6e567435fd5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d0f0866690a940cbaf20f2b1d65a3732, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/47eb033ec2b844b496ceeb692259984f] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=65.4 K 2023-04-01 14:55:59,482 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a71e0a7100ec4f5687d9529cc987530e, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680360957559 2023-04-01 14:55:59,482 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 421794020ccc48d3ad655d62305d1192, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680360957559 2023-04-01 14:55:59,482 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 99760ef96ea84bbcb9c7d41cdbe4042c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680360957610 2023-04-01 14:55:59,482 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 5846d9a730ed4d5a916f1e8431b77c1d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680360957610 2023-04-01 14:55:59,482 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2121bdafd88a4715b474562be5eee12a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680360957765 2023-04-01 14:55:59,483 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 6b95981e596643cbafc2e6e567435fd5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680360957765 2023-04-01 14:55:59,483 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a94633b81cc649ed847372a0edb3cdc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680360958425 2023-04-01 14:55:59,483 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d0f0866690a940cbaf20f2b1d65a3732, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680360958425 2023-04-01 14:55:59,483 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a56a7367b25c4a8fad4a566ece309535, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680360959066 2023-04-01 14:55:59,483 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 47eb033ec2b844b496ceeb692259984f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680360959066 2023-04-01 14:55:59,493 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#A#compaction#512 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:59,493 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#513 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:59,532 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:59,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:59,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:59,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:59,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:59,544 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:59,545 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:59,545 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:59,545 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:59,545 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,545 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:59,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:59,547 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,548 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,548 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:59,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:59,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:59,548 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:59,548 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:59,552 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 14:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:59,552 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:59,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:59,560 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:59,566 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/c580be18480b417c8ff185824702215b 2023-04-01 14:55:59,581 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/32697aac1d54495bb2303d2dd4c4db31 2023-04-01 14:55:59,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361019580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,593 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/3ddeac8365fb45448c3f251e7789b3c0 2023-04-01 14:55:59,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361019585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361019585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361019587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,598 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/c580be18480b417c8ff185824702215b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c580be18480b417c8ff185824702215b 2023-04-01 14:55:59,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361019587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,602 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c580be18480b417c8ff185824702215b, entries=150, sequenceid=564, filesize=11.9 K 2023-04-01 14:55:59,602 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/32697aac1d54495bb2303d2dd4c4db31 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/32697aac1d54495bb2303d2dd4c4db31 2023-04-01 14:55:59,605 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/32697aac1d54495bb2303d2dd4c4db31, entries=150, sequenceid=564, filesize=11.9 K 2023-04-01 14:55:59,606 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/3ddeac8365fb45448c3f251e7789b3c0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/3ddeac8365fb45448c3f251e7789b3c0 2023-04-01 14:55:59,611 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/3ddeac8365fb45448c3f251e7789b3c0, entries=150, sequenceid=564, filesize=11.9 K 2023-04-01 14:55:59,611 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 59ms, sequenceid=564, compaction requested=true 2023-04-01 14:55:59,611 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:59,611 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:59,611 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:59,611 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:59,611 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:59,612 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:59,613 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,613 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,613 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,613 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:59,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ebf8ee8[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:59,615 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:59,615 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,617 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:59,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:59,617 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:59,617 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,617 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:59,618 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,618 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:59,618 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,618 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:59,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:59,621 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:59,621 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:59,621 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:59,621 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:59,621 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:59,629 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,630 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,630 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:59,630 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,633 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,636 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,636 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,636 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,636 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,636 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,636 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:59,636 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:59,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:59,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:59,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:59,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:59,636 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:59,636 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:59,636 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,636 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,636 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:59,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:59,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:59,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 14:55:59,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:59,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:59,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:59,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:59,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:59,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:55:59,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/d6ba749048e548249025bc07d84d93eb 2023-04-01 14:55:59,732 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/1128c06117a04071ada5405c1b78179e 2023-04-01 14:55:59,738 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:59,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:59,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:59,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:59,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:59,742 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:59,742 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:59,742 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:59,742 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:59,743 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,743 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,745 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,745 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:59,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:59,746 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/4b84357ac8d84b83b51fd3b4d0aeb879 2023-04-01 14:55:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:59,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:59,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:59,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:59,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,749 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:59,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361019731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,750 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:59,751 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/d6ba749048e548249025bc07d84d93eb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d6ba749048e548249025bc07d84d93eb 2023-04-01 14:55:59,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,751 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:59,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361019732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,753 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:59,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:59,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:59,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361019739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361019739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361019739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,755 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,755 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d6ba749048e548249025bc07d84d93eb, entries=200, sequenceid=580, filesize=14.3 K 2023-04-01 14:55:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/1128c06117a04071ada5405c1b78179e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1128c06117a04071ada5405c1b78179e 2023-04-01 14:55:59,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:59,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c012b22[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:59,760 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:59,760 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1128c06117a04071ada5405c1b78179e, entries=150, sequenceid=580, filesize=11.9 K 2023-04-01 14:55:59,761 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:59,762 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,762 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:59,762 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/4b84357ac8d84b83b51fd3b4d0aeb879 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4b84357ac8d84b83b51fd3b4d0aeb879 2023-04-01 14:55:59,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,763 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:59,763 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:59,763 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4b84357ac8d84b83b51fd3b4d0aeb879, entries=150, sequenceid=580, filesize=11.9 K 2023-04-01 14:55:59,766 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 67ms, sequenceid=580, compaction requested=true 2023-04-01 14:55:59,766 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:59,767 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 14:55:59,767 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-01 14:55:59,767 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-01 14:55:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:59,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:59,767 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:59,767 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:59,767 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:59,767 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:59,767 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:59,774 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,775 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,775 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,777 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:59,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:59,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:59,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:59,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:59,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:59,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:59,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:59,780 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:59,780 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:59,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:55:59,863 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:55:59,864 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:55:59,864 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:55:59,864 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:55:59,864 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:55:59,864 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:55:59,864 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:55:59,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/6620d59398614225a6994f29e35a6031 2023-04-01 14:55:59,880 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:55:59,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:55:59,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:55:59,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:55:59,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:55:59,888 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:55:59,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:59,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:55:59,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:55:59,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,890 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:59,890 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:59,890 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,890 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:55:59,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:55:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:55:59,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:55:59,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:55:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:55:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:55:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:55:59,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361019884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361019884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,910 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:55:59,910 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:55:59,910 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:59,910 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:55:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361019896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,910 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:55:59,910 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:55:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:55:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:55:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:55:59,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/7c6c5b3edac64718ad64acc9eaf23737 2023-04-01 14:55:59,912 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,914 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:55:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:55:59,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62cc928c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:55:59,917 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:55:59,917 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:55:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:55:59,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:55:59,920 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,920 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:55:59,921 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/b7c83d7fe3944adc81903e6e50fb249c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b7c83d7fe3944adc81903e6e50fb249c 2023-04-01 14:55:59,922 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,922 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:55:59,922 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:59,922 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:55:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:55:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:55:59,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:55:59,926 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:55:59,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:55:59,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:55:59,926 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:55:59,926 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:55:59,929 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,929 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,930 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:59,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:59,930 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into b7c83d7fe3944adc81903e6e50fb249c(size=13.2 K), total size for store is 37.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:59,930 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:59,930 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=11, startTime=1680360959479; duration=0sec 2023-04-01 14:55:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,930 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-01 14:55:59,931 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 5 compacting, 2 eligible, 16 blocking 2023-04-01 14:55:59,931 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,932 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-01 14:55:59,932 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-01 14:55:59,932 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. because compaction request was cancelled 2023-04-01 14:55:59,932 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-01 14:55:59,933 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:55:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:55:59,934 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,934 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,934 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:55:59,934 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:55:59,934 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:55:59,934 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:55:59,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:55:59,934 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:55:59,934 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,934 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:55:59,934 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:55:59,934 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:55:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:55:59,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:55:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:55:59,934 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86426 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-01 14:55:59,934 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:55:59,935 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/C is initiating minor compaction (all files) 2023-04-01 14:55:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:55:59,935 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/C in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:59,935 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/79b115cc0e7f4781a17c8ae9551c518d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b6cebcf535b246808cb9e87c780e550a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7ab321cee76f469bab1907b60208900d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/05819d7750174df2ac2c9a3d25a12ab8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f512032209a34a22b9e3a98e1682afd3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/3ddeac8365fb45448c3f251e7789b3c0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4b84357ac8d84b83b51fd3b4d0aeb879] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=84.4 K 2023-04-01 14:55:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:55:59,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:55:59,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:55:59,935 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 79b115cc0e7f4781a17c8ae9551c518d, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680360957559 2023-04-01 14:55:59,936 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b6cebcf535b246808cb9e87c780e550a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680360957610 2023-04-01 14:55:59,937 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 7ab321cee76f469bab1907b60208900d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680360957765 2023-04-01 14:55:59,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:55:59,937 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 05819d7750174df2ac2c9a3d25a12ab8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680360958425 2023-04-01 14:55:59,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361019903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,937 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f512032209a34a22b9e3a98e1682afd3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680360959066 2023-04-01 14:55:59,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:55:59,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361019904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:55:59,938 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3ddeac8365fb45448c3f251e7789b3c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1680360959440 2023-04-01 14:55:59,938 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 4b84357ac8d84b83b51fd3b4d0aeb879, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680360959585 2023-04-01 14:55:59,943 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/c368ddf486b34008b65b9f6d54336a3f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c368ddf486b34008b65b9f6d54336a3f 2023-04-01 14:55:59,947 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/A of 2ddd6fcff99c4aa857940e9dc8e40f15 into c368ddf486b34008b65b9f6d54336a3f(size=13.2 K), total size for store is 39.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:55:59,947 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:55:59,947 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/A, priority=11, startTime=1680360959479; duration=0sec 2023-04-01 14:55:59,947 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 14:55:59,947 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 7 compacting, 0 eligible, 16 blocking 2023-04-01 14:55:59,947 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-01 14:55:59,947 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-01 14:55:59,947 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. because compaction request was cancelled 2023-04-01 14:55:59,947 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:55:59,949 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:55:59,949 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:55:59,949 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:55:59,949 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b7c83d7fe3944adc81903e6e50fb249c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/32697aac1d54495bb2303d2dd4c4db31, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1128c06117a04071ada5405c1b78179e] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=37.0 K 2023-04-01 14:55:59,949 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting b7c83d7fe3944adc81903e6e50fb249c, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680360959066 2023-04-01 14:55:59,949 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 32697aac1d54495bb2303d2dd4c4db31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1680360959440 2023-04-01 14:55:59,950 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 1128c06117a04071ada5405c1b78179e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680360959585 2023-04-01 14:55:59,960 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#C#compaction#523 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:59,961 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/7110aa52983a41f382b75bdad264b10f 2023-04-01 14:55:59,963 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#524 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:55:59,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/6620d59398614225a6994f29e35a6031 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6620d59398614225a6994f29e35a6031 2023-04-01 14:55:59,975 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6620d59398614225a6994f29e35a6031, entries=350, sequenceid=601, filesize=21.4 K 2023-04-01 14:55:59,976 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/7c6c5b3edac64718ad64acc9eaf23737 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7c6c5b3edac64718ad64acc9eaf23737 2023-04-01 14:55:59,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7c6c5b3edac64718ad64acc9eaf23737, entries=150, sequenceid=601, filesize=11.9 K 2023-04-01 14:55:59,981 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/7110aa52983a41f382b75bdad264b10f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7110aa52983a41f382b75bdad264b10f 2023-04-01 14:55:59,997 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/0ef8c263721a4867927e89a951b5171a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/0ef8c263721a4867927e89a951b5171a 2023-04-01 14:56:00,005 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7110aa52983a41f382b75bdad264b10f, entries=150, sequenceid=601, filesize=11.9 K 2023-04-01 14:56:00,005 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 142ms, sequenceid=601, compaction requested=true 2023-04-01 14:56:00,006 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:00,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:00,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 14:56:00,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-01 14:56:00,009 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/C of 2ddd6fcff99c4aa857940e9dc8e40f15 into 0ef8c263721a4867927e89a951b5171a(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:00,010 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:00,010 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/C, priority=9, startTime=1680360959767; duration=0sec 2023-04-01 14:56:00,010 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-01 14:56:00,010 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:56:00,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:56:00,011 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 14:56:00,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:56:00,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:00,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:56:00,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:00,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:56:00,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:00,012 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 62276 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:56:00,012 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/A is initiating minor compaction (all files) 2023-04-01 14:56:00,012 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/A in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,012 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c368ddf486b34008b65b9f6d54336a3f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c580be18480b417c8ff185824702215b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d6ba749048e548249025bc07d84d93eb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6620d59398614225a6994f29e35a6031] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=60.8 K 2023-04-01 14:56:00,013 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c368ddf486b34008b65b9f6d54336a3f, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680360959066 2023-04-01 14:56:00,013 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c580be18480b417c8ff185824702215b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1680360959440 2023-04-01 14:56:00,013 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting d6ba749048e548249025bc07d84d93eb, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680360959584 2023-04-01 14:56:00,014 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6620d59398614225a6994f29e35a6031, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680360959728 2023-04-01 14:56:00,022 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#A#compaction#526 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:00,032 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/3cfdead8adbd4b05b826b6e77310406e 2023-04-01 14:56:00,035 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:00,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:00,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:00,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:00,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:00,052 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:00,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:00,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:00,052 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:00,053 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,053 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,054 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:00,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,055 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:00,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:00,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:00,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:00,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:00,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:00,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:00,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:00,060 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:00,060 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,060 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:56:00,061 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:56:00,061 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,061 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:00,061 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:00,061 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,066 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:00,066 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:00,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361020056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361020057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:00,069 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:00,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361020058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361020057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,070 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:00,070 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@551cbfc6[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:00,072 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:00,072 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,073 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:00,074 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,074 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:00,076 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:00,076 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:00,076 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,076 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:00,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:00,078 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:00,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:00,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:00,078 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:00,078 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:00,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361020069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,082 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,082 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,082 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:00,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:00,082 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:00,084 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,086 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:00,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:00,086 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,086 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:00,087 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:00,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:00,087 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:00,086 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:00,087 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:00,087 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:00,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:00,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361020169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361020169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361020170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361020170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361020180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,188 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:00,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:00,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:00,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:00,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:00,192 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:00,192 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:00,192 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:00,192 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:00,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:00,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:00,194 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,195 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:00,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:00,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:00,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:56:00,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:56:00,196 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:00,197 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:00,199 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,199 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:00,200 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,200 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1796a343[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:00,201 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:00,201 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,203 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:00,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,203 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:00,203 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:00,203 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:00,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:00,205 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,205 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:00,205 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:00,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:00,207 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:00,207 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:00,207 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:00,207 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:00,207 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:00,209 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:00,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,211 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,212 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,212 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:00,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:00,213 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:00,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,221 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:00,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:00,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:00,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,221 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:00,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:00,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:00,221 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:00,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:00,322 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:00,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:00,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:00,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:00,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:00,325 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:00,325 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:00,325 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:00,325 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:00,326 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,326 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,327 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,327 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:00,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:00,328 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,328 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:00,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:00,329 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:00,329 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:00,329 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:00,329 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:00,330 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:00,330 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,330 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:56:00,330 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:56:00,332 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:00,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:00,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:00,332 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:00,334 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,334 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:00,335 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,335 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dbd2a8[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:00,336 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:00,336 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,339 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:00,339 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:00,339 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:00,340 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,341 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:00,341 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:00,341 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:00,342 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:00,342 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:00,342 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:00,342 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:00,342 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:00,342 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:00,342 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:00,344 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:00,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:00,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:00,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,346 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,347 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,349 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:00,350 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:00,350 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:00,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:00,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:00,350 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,350 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,350 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,350 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,350 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:00,350 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:00,350 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:00,350 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,350 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:00,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:00,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361020375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361020375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361020376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361020376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361020385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,400 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/2412accd4aa5422188a02a54af90925c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2412accd4aa5422188a02a54af90925c 2023-04-01 14:56:00,405 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into 2412accd4aa5422188a02a54af90925c(size=13.3 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:00,405 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:00,405 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=13, startTime=1680360959767; duration=0sec 2023-04-01 14:56:00,405 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 14:56:00,405 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-01 14:56:00,406 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-01 14:56:00,406 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-01 14:56:00,406 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. because compaction request was cancelled 2023-04-01 14:56:00,406 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-01 14:56:00,407 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-01 14:56:00,407 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-01 14:56:00,407 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. because compaction request was cancelled 2023-04-01 14:56:00,450 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:00,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:00,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:00,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:00,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:00,454 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:00,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:00,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:00,454 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:00,455 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,455 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,457 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:00,457 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,458 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,458 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:00,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:00,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:00,460 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:00,460 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/5a003de23c0846ba8b3eb7c38db98dd4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5a003de23c0846ba8b3eb7c38db98dd4 2023-04-01 14:56:00,460 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,460 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:56:00,460 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:56:00,460 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:00,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:00,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:00,461 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:00,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,462 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:00,464 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,464 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:00,468 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/A of 2ddd6fcff99c4aa857940e9dc8e40f15 into 5a003de23c0846ba8b3eb7c38db98dd4(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:00,468 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:00,468 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/A, priority=12, startTime=1680360960006; duration=0sec 2023-04-01 14:56:00,468 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:00,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:00,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@485688d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:00,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:00,468 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,469 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/0d0e6d56c7a54bbc95a990c54fd36f6c 2023-04-01 14:56:00,471 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:00,471 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,471 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:00,473 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:00,473 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:00,473 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:00,473 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:00,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:00,476 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:00,476 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:00,476 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:00,476 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:00,476 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:00,479 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,479 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,479 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:00,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,479 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:00,483 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,485 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:00,485 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:00,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:00,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:00,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:00,485 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,485 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:00,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:00,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:00,486 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:00,486 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:00,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:00,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:00,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:00,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/854c39466248448d8269e14fad11a13c 2023-04-01 14:56:00,507 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/3cfdead8adbd4b05b826b6e77310406e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3cfdead8adbd4b05b826b6e77310406e 2023-04-01 14:56:00,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3cfdead8adbd4b05b826b6e77310406e, entries=200, sequenceid=617, filesize=14.3 K 2023-04-01 14:56:00,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/0d0e6d56c7a54bbc95a990c54fd36f6c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/0d0e6d56c7a54bbc95a990c54fd36f6c 2023-04-01 14:56:00,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/0d0e6d56c7a54bbc95a990c54fd36f6c, entries=150, sequenceid=617, filesize=11.9 K 2023-04-01 14:56:00,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/854c39466248448d8269e14fad11a13c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/854c39466248448d8269e14fad11a13c 2023-04-01 14:56:00,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/854c39466248448d8269e14fad11a13c, entries=150, sequenceid=617, filesize=11.9 K 2023-04-01 14:56:00,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 516ms, sequenceid=617, compaction requested=true 2023-04-01 14:56:00,527 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:00,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:00,527 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-01 14:56:00,527 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:56:00,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:00,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:00,528 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-01 14:56:00,528 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-01 14:56:00,528 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. because compaction request was cancelled 2023-04-01 14:56:00,528 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:56:00,528 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:56:00,529 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:56:00,529 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,529 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2412accd4aa5422188a02a54af90925c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7c6c5b3edac64718ad64acc9eaf23737, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/0d0e6d56c7a54bbc95a990c54fd36f6c] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=37.1 K 2023-04-01 14:56:00,529 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2412accd4aa5422188a02a54af90925c, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680360959585 2023-04-01 14:56:00,530 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 7c6c5b3edac64718ad64acc9eaf23737, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680360959861 2023-04-01 14:56:00,530 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:56:00,530 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/C is initiating minor compaction (all files) 2023-04-01 14:56:00,530 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/C in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,530 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/0ef8c263721a4867927e89a951b5171a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7110aa52983a41f382b75bdad264b10f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/854c39466248448d8269e14fad11a13c] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=37.0 K 2023-04-01 14:56:00,530 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 0d0e6d56c7a54bbc95a990c54fd36f6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1680360959882 2023-04-01 14:56:00,531 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 0ef8c263721a4867927e89a951b5171a, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680360959585 2023-04-01 14:56:00,534 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 7110aa52983a41f382b75bdad264b10f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680360959861 2023-04-01 14:56:00,534 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 854c39466248448d8269e14fad11a13c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1680360959882 2023-04-01 14:56:00,541 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#C#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:00,544 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#530 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:00,586 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:00,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:00,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:00,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:00,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:00,590 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:00,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:00,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:00,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:00,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,590 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,591 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:00,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:00,591 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,592 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,592 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:00,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:00,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:00,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:00,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:00,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:56:00,594 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:56:00,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:56:00,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:00,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:56:00,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:00,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:56:00,594 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:00,610 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/1d471440c75f43a09f37b52b0e1d6daf 2023-04-01 14:56:00,630 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/ecb753aa3f744c6ea76a91d01f9f685b 2023-04-01 14:56:00,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:56:00,683 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:56:00,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361020692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361020693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361020694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361020698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361020699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361020800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361020800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361020800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361020801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:00,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361020806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:00,959 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/3bb990bdc4254dcf83136ce0d9b5141f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/3bb990bdc4254dcf83136ce0d9b5141f 2023-04-01 14:56:00,963 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/C of 2ddd6fcff99c4aa857940e9dc8e40f15 into 3bb990bdc4254dcf83136ce0d9b5141f(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:00,964 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:00,964 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/C, priority=13, startTime=1680360960527; duration=0sec 2023-04-01 14:56:00,964 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:00,973 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/2420216877df4ebe85751bb8de295643 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2420216877df4ebe85751bb8de295643 2023-04-01 14:56:00,982 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into 2420216877df4ebe85751bb8de295643(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:00,982 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:00,982 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=13, startTime=1680360960527; duration=0sec 2023-04-01 14:56:00,982 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:01,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361021004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361021004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361021004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361021010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361021013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,052 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/5d56b15dc28e409f991e639c71d3e7b1 2023-04-01 14:56:01,058 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/1d471440c75f43a09f37b52b0e1d6daf as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/1d471440c75f43a09f37b52b0e1d6daf 2023-04-01 14:56:01,063 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/1d471440c75f43a09f37b52b0e1d6daf, entries=150, sequenceid=638, filesize=11.9 K 2023-04-01 14:56:01,064 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/ecb753aa3f744c6ea76a91d01f9f685b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ecb753aa3f744c6ea76a91d01f9f685b 2023-04-01 14:56:01,068 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ecb753aa3f744c6ea76a91d01f9f685b, entries=150, sequenceid=638, filesize=11.9 K 2023-04-01 14:56:01,071 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/5d56b15dc28e409f991e639c71d3e7b1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/5d56b15dc28e409f991e639c71d3e7b1 2023-04-01 14:56:01,075 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/5d56b15dc28e409f991e639c71d3e7b1, entries=150, sequenceid=638, filesize=11.9 K 2023-04-01 14:56:01,076 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 482ms, sequenceid=638, compaction requested=true 2023-04-01 14:56:01,076 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:01,076 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:01,078 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:01,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34571554[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:01,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:01,080 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,083 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,083 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:01,083 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:01,083 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:01,083 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,083 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:01,084 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,084 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:01,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:01,087 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:01,087 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:01,087 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:01,087 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:01,087 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:01,088 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,088 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:01,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:01,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:01,089 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:01,092 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,097 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:01,097 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:01,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:01,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:01,097 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,098 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:01,098 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:01,098 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,098 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,098 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,098 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,097 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:01,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:01,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:01,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:01,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:01,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:01,098 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:01,098 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:01,198 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:01,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:01,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:01,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:01,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:01,202 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:01,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:01,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:01,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:01,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,202 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:01,204 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,204 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:01,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:01,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:01,204 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,204 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:01,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:01,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:01,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:01,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:01,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:01,205 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:01,205 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:01,206 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:01,208 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:01,208 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:01,208 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:01,208 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:56:01,208 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 14:56:01,209 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:56:01,209 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:01,209 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:56:01,209 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:01,209 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:56:01,209 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:01,239 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/2a6aba8865dd4cd683f9f90fa4514007 2023-04-01 14:56:01,264 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/8675329f57b6440fbe014d9e5c12d741 2023-04-01 14:56:01,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:56:01,312 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:56:01,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361021330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361021331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361021335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361021336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361021337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361021438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361021438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361021447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361021447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361021448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361021644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361021644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361021654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361021654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361021655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,682 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/7c11070ae57d47e6b478b9388359ea61 2023-04-01 14:56:01,687 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/2a6aba8865dd4cd683f9f90fa4514007 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2a6aba8865dd4cd683f9f90fa4514007 2023-04-01 14:56:01,694 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2a6aba8865dd4cd683f9f90fa4514007, entries=150, sequenceid=653, filesize=11.9 K 2023-04-01 14:56:01,695 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/8675329f57b6440fbe014d9e5c12d741 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8675329f57b6440fbe014d9e5c12d741 2023-04-01 14:56:01,706 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8675329f57b6440fbe014d9e5c12d741, entries=150, sequenceid=653, filesize=11.9 K 2023-04-01 14:56:01,706 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/7c11070ae57d47e6b478b9388359ea61 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7c11070ae57d47e6b478b9388359ea61 2023-04-01 14:56:01,710 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7c11070ae57d47e6b478b9388359ea61, entries=150, sequenceid=653, filesize=11.9 K 2023-04-01 14:56:01,711 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 503ms, sequenceid=653, compaction requested=true 2023-04-01 14:56:01,711 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:01,711 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:01,711 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:01,711 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:01,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:01,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:01,713 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:01,714 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,714 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:01,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:01,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f93426[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:01,716 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:01,717 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,720 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:01,720 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:01,720 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:01,720 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,720 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:01,722 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:01,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:01,722 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:01,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:01,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:01,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:01,726 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:01,726 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:01,726 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:01,726 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:01,726 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:01,727 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,727 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:01,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,727 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:01,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:01,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:01,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:01,728 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:01,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:01,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,730 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:01,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:01,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,732 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:01,732 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:01,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:01,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:01,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:01,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:01,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:01,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:01,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:01,733 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:01,733 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:01,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:01,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:01,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:01,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:01,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,833 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:01,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:01,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:01,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:01,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:01,838 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:01,838 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:01,838 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:01,838 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:01,839 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,839 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:01,845 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:01,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:01,846 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:01,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:01,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:01,846 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,847 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:01,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:01,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:01,852 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:01,852 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:01,852 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:01,853 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:01,853 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:01,854 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:01,854 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:56:01,854 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:56:01,856 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:56:01,856 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:01,856 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:56:01,856 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:01,856 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:56:01,856 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:01,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:56:01,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:56:01,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361021967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361021968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361021969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361021971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:01,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:01,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361021972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361022076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361022077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361022078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361022079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361022080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,271 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/e9d75e3761a64032aab673123d558473 2023-04-01 14:56:02,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361022285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361022285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361022286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361022286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361022292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,300 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/e32dbdf7f07f4802b7c1a49f9b8f20a4 2023-04-01 14:56:02,332 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/00cb530961374a47afe3611e6e1eee74 2023-04-01 14:56:02,337 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/e9d75e3761a64032aab673123d558473 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e9d75e3761a64032aab673123d558473 2023-04-01 14:56:02,341 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e9d75e3761a64032aab673123d558473, entries=150, sequenceid=672, filesize=11.9 K 2023-04-01 14:56:02,341 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/e32dbdf7f07f4802b7c1a49f9b8f20a4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e32dbdf7f07f4802b7c1a49f9b8f20a4 2023-04-01 14:56:02,347 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e32dbdf7f07f4802b7c1a49f9b8f20a4, entries=150, sequenceid=672, filesize=11.9 K 2023-04-01 14:56:02,348 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/00cb530961374a47afe3611e6e1eee74 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/00cb530961374a47afe3611e6e1eee74 2023-04-01 14:56:02,361 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/00cb530961374a47afe3611e6e1eee74, entries=150, sequenceid=672, filesize=11.9 K 2023-04-01 14:56:02,362 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 508ms, sequenceid=672, compaction requested=true 2023-04-01 14:56:02,362 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:02,362 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:02,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:02,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:02,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:02,362 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:02,364 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,365 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,365 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:02,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aebfac3[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:02,367 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:02,367 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,369 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:02,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:02,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:02,370 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,370 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:02,371 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:02,371 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:02,371 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:02,371 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:02,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:02,375 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:02,375 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:02,375 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:02,375 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:02,375 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:02,378 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,378 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:02,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,378 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:02,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:02,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,380 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,383 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,385 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:02,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:02,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:02,386 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,386 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,386 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,386 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,386 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,386 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,386 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:02,386 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:02,386 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:02,385 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:02,386 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:02,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:02,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:02,486 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:02,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:02,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:02,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:02,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:02,489 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:02,489 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:02,490 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:02,490 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:02,490 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,490 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,491 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,491 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:02,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:02,492 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,492 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:02,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:02,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:02,493 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:02,493 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:02,493 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:02,493 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:02,493 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:02,493 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:02,494 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:56:02,494 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 14:56:02,494 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:56:02,494 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:02,494 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:56:02,494 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:02,494 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:56:02,494 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:02,505 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/aef386d3ed944fba89ca6708467e4bed 2023-04-01 14:56:02,522 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/a8c37a76ee9540b1b1682c1c62607f02 2023-04-01 14:56:02,533 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/91eb5660697840609cb8983ed3c6ba13 2023-04-01 14:56:02,538 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/aef386d3ed944fba89ca6708467e4bed as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aef386d3ed944fba89ca6708467e4bed 2023-04-01 14:56:02,541 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aef386d3ed944fba89ca6708467e4bed, entries=150, sequenceid=685, filesize=11.9 K 2023-04-01 14:56:02,542 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/a8c37a76ee9540b1b1682c1c62607f02 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a8c37a76ee9540b1b1682c1c62607f02 2023-04-01 14:56:02,549 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a8c37a76ee9540b1b1682c1c62607f02, entries=150, sequenceid=685, filesize=11.9 K 2023-04-01 14:56:02,550 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/91eb5660697840609cb8983ed3c6ba13 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/91eb5660697840609cb8983ed3c6ba13 2023-04-01 14:56:02,553 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/91eb5660697840609cb8983ed3c6ba13, entries=150, sequenceid=685, filesize=11.9 K 2023-04-01 14:56:02,554 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 60ms, sequenceid=685, compaction requested=true 2023-04-01 14:56:02,554 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:02,554 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:02,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:02,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:02,554 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:02,556 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,557 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,557 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65852213[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:02,561 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:02,561 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,564 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,565 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:02,565 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:02,565 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:02,566 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,566 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:02,567 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:02,567 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,567 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,567 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:02,571 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:02,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:02,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:02,571 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:02,571 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:02,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:02,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,575 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,578 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:02,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:02,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:02,582 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:02,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:02,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:02,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,582 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:02,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:02,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:02,582 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:02,582 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:02,583 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:56:02,607 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-01 14:56:02,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:56:02,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:02,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:56:02,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:02,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:56:02,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:02,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/86d22adc6de6473895e2ec3b71b4edf0 2023-04-01 14:56:02,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/3b1a9f9987e54e83b73ba4be5cf53bf4 2023-04-01 14:56:02,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360962624, max timestamp=1680360962643Num uniques 859; 2023-04-01 14:56:02,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:02,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361022639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360962624, max timestamp=1680360962643Num uniques 859; 2023-04-01 14:56:02,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:02,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360962624, max timestamp=1680360962643Num uniques 859; 2023-04-01 14:56:02,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:02,683 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:02,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:02,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:02,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:02,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:02,686 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:02,686 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:02,686 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:02,686 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:02,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,689 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:02,689 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:02,689 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,689 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:02,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:02,691 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:02,691 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:56:02,691 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:56:02,691 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:02,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361022675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:02,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361022685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,695 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,695 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:02,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69746642[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:02,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:02,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,699 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:02,699 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,699 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:02,700 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:02,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,700 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:02,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:02,703 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:02,703 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:02,703 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:02,703 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:02,703 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:02,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361022690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361022692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,706 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,706 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:02,706 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:02,709 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,711 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:02,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:02,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:02,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,713 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:02,713 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:02,713 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:02,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:02,713 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:02,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:02,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361022748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361022793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361022793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,813 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:02,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:02,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361022806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361022806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:02,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:02,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:02,817 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:02,817 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:02,817 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:02,818 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:02,818 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,818 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,819 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:02,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:02,819 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,820 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,820 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:02,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:02,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:02,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:02,821 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:02,821 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:02,821 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:02,821 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:02,821 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:02,821 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:02,821 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:56:02,822 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:56:02,822 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:02,822 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:02,822 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:02,822 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,822 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:02,822 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:02,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,823 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,824 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,824 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:02,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3673374e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:02,825 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:02,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,827 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,828 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:02,828 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:02,828 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:02,833 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,833 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:02,834 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:02,834 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:02,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:02,838 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:02,838 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:02,838 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:02,838 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:02,838 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:02,840 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,840 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:02,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,840 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,841 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,842 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,844 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:02,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:02,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:02,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:02,844 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:02,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:02,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:02,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:02,845 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:02,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:02,945 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:02,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:02,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:02,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:02,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:02,948 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:02,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:02,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:02,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:02,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,949 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,952 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,952 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:02,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:02,952 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,952 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:02,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:02,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:02,954 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:02,954 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. started... 2023-04-01 14:56:02,954 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. as already flushing 2023-04-01 14:56:02,954 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:02,956 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,957 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,957 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23dbeea1[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:02,959 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:02,960 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,961 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:02,961 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,961 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:02,962 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,962 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,962 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:02,963 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:02,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48086 deadline: 1680361022956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:02,967 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:02,967 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:02,967 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:02,967 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:02,967 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:02,970 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,970 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:02,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,970 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:02,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:02,971 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,976 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,976 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,976 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:02,976 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:02,976 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:02,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:02,977 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:02,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:02,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:02,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:02,976 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:02,977 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,977 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:02,977 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:02,977 DEBUG [Thread-3059] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:02,977 DEBUG [Thread-3059] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:02,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:02,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:02,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:03,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:03,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48032 deadline: 1680361023004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:03,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:03,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48080 deadline: 1680361023004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:03,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:03,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48036 deadline: 1680361023016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:03,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:03,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48116 deadline: 1680361023016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:03,050 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/59ba248de4154ed58a030e72b73c3633 2023-04-01 14:56:03,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/86d22adc6de6473895e2ec3b71b4edf0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/86d22adc6de6473895e2ec3b71b4edf0 2023-04-01 14:56:03,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/86d22adc6de6473895e2ec3b71b4edf0, entries=200, sequenceid=695, filesize=14.3 K 2023-04-01 14:56:03,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/3b1a9f9987e54e83b73ba4be5cf53bf4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/3b1a9f9987e54e83b73ba4be5cf53bf4 2023-04-01 14:56:03,060 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/3b1a9f9987e54e83b73ba4be5cf53bf4, entries=100, sequenceid=695, filesize=9.5 K 2023-04-01 14:56:03,061 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/59ba248de4154ed58a030e72b73c3633 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/59ba248de4154ed58a030e72b73c3633 2023-04-01 14:56:03,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/59ba248de4154ed58a030e72b73c3633, entries=100, sequenceid=695, filesize=9.5 K 2023-04-01 14:56:03,073 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 466ms, sequenceid=695, compaction requested=true 2023-04-01 14:56:03,073 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:03,073 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:03,073 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-01 14:56:03,073 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-01 14:56:03,073 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:03,073 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:03,076 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72204 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-01 14:56:03,076 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/B is initiating minor compaction (all files) 2023-04-01 14:56:03,076 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/B in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:03,076 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2420216877df4ebe85751bb8de295643, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ecb753aa3f744c6ea76a91d01f9f685b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8675329f57b6440fbe014d9e5c12d741, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e32dbdf7f07f4802b7c1a49f9b8f20a4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a8c37a76ee9540b1b1682c1c62607f02, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/3b1a9f9987e54e83b73ba4be5cf53bf4] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=70.5 K 2023-04-01 14:56:03,077 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 91642 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-01 14:56:03,077 DEBUG [Thread-3066] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed183a4 to 127.0.0.1:60465 2023-04-01 14:56:03,077 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): 2ddd6fcff99c4aa857940e9dc8e40f15/A is initiating minor compaction (all files) 2023-04-01 14:56:03,077 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ddd6fcff99c4aa857940e9dc8e40f15/A in TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:03,077 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5a003de23c0846ba8b3eb7c38db98dd4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3cfdead8adbd4b05b826b6e77310406e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/1d471440c75f43a09f37b52b0e1d6daf, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2a6aba8865dd4cd683f9f90fa4514007, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e9d75e3761a64032aab673123d558473, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aef386d3ed944fba89ca6708467e4bed, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/86d22adc6de6473895e2ec3b71b4edf0] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp, totalSize=89.5 K 2023-04-01 14:56:03,077 DEBUG [Thread-3059] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:03,077 DEBUG [Thread-3066] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:03,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:03,077 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 5a003de23c0846ba8b3eb7c38db98dd4, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680360959861 2023-04-01 14:56:03,077 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2420216877df4ebe85751bb8de295643, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1680360959882 2023-04-01 14:56:03,079 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 3cfdead8adbd4b05b826b6e77310406e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1680360959882 2023-04-01 14:56:03,079 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ecb753aa3f744c6ea76a91d01f9f685b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1680360960042 2023-04-01 14:56:03,079 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 1d471440c75f43a09f37b52b0e1d6daf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1680360960042 2023-04-01 14:56:03,080 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 8675329f57b6440fbe014d9e5c12d741, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1680360960691 2023-04-01 14:56:03,080 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 2a6aba8865dd4cd683f9f90fa4514007, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1680360960691 2023-04-01 14:56:03,080 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e32dbdf7f07f4802b7c1a49f9b8f20a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1680360961322 2023-04-01 14:56:03,080 DEBUG [Thread-3062] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:60465 2023-04-01 14:56:03,080 DEBUG [Thread-3062] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:03,081 DEBUG [Thread-3064] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:60465 2023-04-01 14:56:03,081 DEBUG [Thread-3064] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:03,081 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a8c37a76ee9540b1b1682c1c62607f02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1680360961966 2023-04-01 14:56:03,082 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting e9d75e3761a64032aab673123d558473, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1680360961322 2023-04-01 14:56:03,082 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3b1a9f9987e54e83b73ba4be5cf53bf4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1680360962604 2023-04-01 14:56:03,082 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting aef386d3ed944fba89ca6708467e4bed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1680360961966 2023-04-01 14:56:03,082 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 86d22adc6de6473895e2ec3b71b4edf0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=695, earliestPutTs=1680360962599 2023-04-01 14:56:03,089 DEBUG [Thread-3060] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63093e21 to 127.0.0.1:60465 2023-04-01 14:56:03,089 DEBUG [Thread-3060] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:03,092 DEBUG [Thread-3068] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10a344e6 to 127.0.0.1:60465 2023-04-01 14:56:03,092 DEBUG [Thread-3068] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:03,095 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#A#compaction#547 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:03,095 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ddd6fcff99c4aa857940e9dc8e40f15#B#compaction#546 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:03,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:56:03,267 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-01 14:56:03,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:56:03,267 DEBUG [Thread-3055] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:60465 2023-04-01 14:56:03,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:03,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:56:03,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:03,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:56:03,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:03,267 DEBUG [Thread-3055] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:03,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/422ea860e0f8477894b285bc8b3624cd 2023-04-01 14:56:03,320 DEBUG [Thread-3053] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:60465 2023-04-01 14:56:03,320 DEBUG [Thread-3053] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:03,320 DEBUG [Thread-3049] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ed9117 to 127.0.0.1:60465 2023-04-01 14:56:03,320 DEBUG [Thread-3049] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:03,323 DEBUG [Thread-3051] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50002343 to 127.0.0.1:60465 2023-04-01 14:56:03,323 DEBUG [Thread-3051] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:03,324 DEBUG [Thread-3057] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:60465 2023-04-01 14:56:03,324 DEBUG [Thread-3057] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:03,324 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 14:56:03,324 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-04-01 14:56:03,324 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 126 2023-04-01 14:56:03,324 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 122 2023-04-01 14:56:03,324 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 111 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 115 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 1773 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 5314 rows 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 1778 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 5332 rows 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 1775 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 5322 rows 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 1775 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 5325 rows 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(402): scanned 1769 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(403): verified 5307 rows 2023-04-01 14:56:03,325 INFO [Listener at localhost/46667] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 14:56:03,325 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec943a9 to 127.0.0.1:60465 2023-04-01 14:56:03,325 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:03,329 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 14:56:03,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-01 14:56:03,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:03,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 14:56:03,333 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360963333"}]},"ts":"1680360963333"} 2023-04-01 14:56:03,334 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 14:56:03,335 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 14:56:03,336 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ddd6fcff99c4aa857940e9dc8e40f15, UNASSIGN}] 2023-04-01 14:56:03,337 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ddd6fcff99c4aa857940e9dc8e40f15, UNASSIGN 2023-04-01 14:56:03,338 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2ddd6fcff99c4aa857940e9dc8e40f15, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:03,338 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360963338"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360963338"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360963338"}]},"ts":"1680360963338"} 2023-04-01 14:56:03,339 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:56:03,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 14:56:03,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:56:03,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2ddd6fcff99c4aa857940e9dc8e40f15, disabling compactions & flushes 2023-04-01 14:56:03,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 2 compactions & cache flush to complete for region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:03,505 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/c1f874c5d15547faaf6bdbc9da305dcb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/c1f874c5d15547faaf6bdbc9da305dcb 2023-04-01 14:56:03,505 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/4684f28f81024ca58a84f3cb39959aa1 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/4684f28f81024ca58a84f3cb39959aa1 2023-04-01 14:56:03,509 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/B of 2ddd6fcff99c4aa857940e9dc8e40f15 into c1f874c5d15547faaf6bdbc9da305dcb(size=13.6 K), total size for store is 13.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:03,509 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2ddd6fcff99c4aa857940e9dc8e40f15/A of 2ddd6fcff99c4aa857940e9dc8e40f15 into 4684f28f81024ca58a84f3cb39959aa1(size=13.6 K), total size for store is 13.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:03,510 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:03,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:03,510 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/B, priority=10, startTime=1680360963073; duration=0sec 2023-04-01 14:56:03,510 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:03,510 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15., storeName=2ddd6fcff99c4aa857940e9dc8e40f15/A, priority=9, startTime=1680360963073; duration=0sec 2023-04-01 14:56:03,510 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:03,510 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:03,510 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. because compaction request was cancelled 2023-04-01 14:56:03,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 14:56:03,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/bc3162184a1d4a62b8ba99dcebaeb30b 2023-04-01 14:56:03,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/514a61ce7f8c4c1dbc86401fc94aa9f4 2023-04-01 14:56:03,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/422ea860e0f8477894b285bc8b3624cd as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/422ea860e0f8477894b285bc8b3624cd 2023-04-01 14:56:03,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/422ea860e0f8477894b285bc8b3624cd, entries=150, sequenceid=732, filesize=11.9 K 2023-04-01 14:56:03,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/bc3162184a1d4a62b8ba99dcebaeb30b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/bc3162184a1d4a62b8ba99dcebaeb30b 2023-04-01 14:56:03,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/bc3162184a1d4a62b8ba99dcebaeb30b, entries=150, sequenceid=732, filesize=11.9 K 2023-04-01 14:56:03,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/514a61ce7f8c4c1dbc86401fc94aa9f4 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/514a61ce7f8c4c1dbc86401fc94aa9f4 2023-04-01 14:56:03,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/514a61ce7f8c4c1dbc86401fc94aa9f4, entries=150, sequenceid=732, filesize=11.9 K 2023-04-01 14:56:03,705 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=26.84 KB/27480 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 438ms, sequenceid=732, compaction requested=true 2023-04-01 14:56:03,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:03,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:03,705 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. because compaction request was cancelled 2023-04-01 14:56:03,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:03,705 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:03,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. after waiting 0 ms 2023-04-01 14:56:03,706 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. because compaction request was cancelled 2023-04-01 14:56:03,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:03,706 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:03,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2ddd6fcff99c4aa857940e9dc8e40f15 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-01 14:56:03,706 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. because compaction request was cancelled 2023-04-01 14:56:03,706 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:03,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=A 2023-04-01 14:56:03,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:03,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=B 2023-04-01 14:56:03,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:03,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ddd6fcff99c4aa857940e9dc8e40f15, store=C 2023-04-01 14:56:03,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:03,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 14:56:04,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=741 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/04a5f812e58043328bac2671a5324f14 2023-04-01 14:56:04,121 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=741 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/acb203d1296941a0bef94c8a372a8511 2023-04-01 14:56:04,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 14:56:04,535 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=741 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/06a5f3f2d1094df994dd9454b488b9aa 2023-04-01 14:56:04,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/A/04a5f812e58043328bac2671a5324f14 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/04a5f812e58043328bac2671a5324f14 2023-04-01 14:56:04,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/04a5f812e58043328bac2671a5324f14, entries=150, sequenceid=741, filesize=11.9 K 2023-04-01 14:56:04,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/B/acb203d1296941a0bef94c8a372a8511 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/acb203d1296941a0bef94c8a372a8511 2023-04-01 14:56:04,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/acb203d1296941a0bef94c8a372a8511, entries=150, sequenceid=741, filesize=11.9 K 2023-04-01 14:56:04,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/.tmp/C/06a5f3f2d1094df994dd9454b488b9aa as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/06a5f3f2d1094df994dd9454b488b9aa 2023-04-01 14:56:04,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/06a5f3f2d1094df994dd9454b488b9aa, entries=150, sequenceid=741, filesize=11.9 K 2023-04-01 14:56:04,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 2ddd6fcff99c4aa857940e9dc8e40f15 in 842ms, sequenceid=741, compaction requested=true 2023-04-01 14:56:04,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3f280e4375b74a8f925245d8cf4b4116, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aa9352d72262437198984120feb40433, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/be361e4697ee498f9a3614abb8773560, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/cd74b407566949008d703f7eec4995e8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7e577242145e42578af6b53d80aa7d31, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e971586a3b24422cad901b3ca1422216, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/220b7d59b99f4e7abd8c089a0adae0b9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/22beee555e2d4cb8ae7be231e0113544, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/544e93c2f6a0459fa91a9f3778efcddf, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5f99100790ab4c13b44647cad6f4ecae, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/632675a61a444c72a87b598725d468d7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3043b40fb6764642bbe3bbaac4629783, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6243b27b51f54a0994a6c481735ad7c1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/b189447fd8ed40dfb8fd0873364365ab, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/57814530ced6486093bccae7f7874f36, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2957beb01ba543c09268d363cb15843b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7a8c7e62321048d9997d7a0a88ad065d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2dc4bd2f94e34f0e932b8d8416205de4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/0d4d8ad2a2fb4426814e51b4faaf069c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/044d8ec440d542c9a380c5dba51a99cb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/edbc0a52a2314d0782c5dfe5e20f514b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c1d09a0d98c44ed6a201ac86819a7c47, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/18a6975879fb41659cacedde8fa6d674, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/a32203114bd2459e84f5395741bdfe49, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d55066b5e4534ca8a1d5468e4083411d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/cc057543248c4f2e95b5055a140210d0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/65d59e3faff14edab5c9a2c901f8a3f7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/68666cc5363b412ea262cb48487bb737, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/00edfd55ca76480a99658f9d320ab91a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/35e38d944ffd4095b108b865c0113a1a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/55220ae246934adbae18621d67723774, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c8a33789e19b4438907fffef3cd5c9d1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/781c9e92594c46a7a44e3c3dad80c50e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/f8a1b9f717ac40ad817807407a402b03, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/421794020ccc48d3ad655d62305d1192, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5846d9a730ed4d5a916f1e8431b77c1d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6b95981e596643cbafc2e6e567435fd5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d0f0866690a940cbaf20f2b1d65a3732, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/47eb033ec2b844b496ceeb692259984f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c368ddf486b34008b65b9f6d54336a3f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c580be18480b417c8ff185824702215b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d6ba749048e548249025bc07d84d93eb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6620d59398614225a6994f29e35a6031, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5a003de23c0846ba8b3eb7c38db98dd4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3cfdead8adbd4b05b826b6e77310406e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/1d471440c75f43a09f37b52b0e1d6daf, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2a6aba8865dd4cd683f9f90fa4514007, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e9d75e3761a64032aab673123d558473, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aef386d3ed944fba89ca6708467e4bed, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/86d22adc6de6473895e2ec3b71b4edf0] to archive 2023-04-01 14:56:04,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:56:04,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3f280e4375b74a8f925245d8cf4b4116 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3f280e4375b74a8f925245d8cf4b4116 2023-04-01 14:56:04,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aa9352d72262437198984120feb40433 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aa9352d72262437198984120feb40433 2023-04-01 14:56:04,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/be361e4697ee498f9a3614abb8773560 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/be361e4697ee498f9a3614abb8773560 2023-04-01 14:56:04,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/cd74b407566949008d703f7eec4995e8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/cd74b407566949008d703f7eec4995e8 2023-04-01 14:56:04,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7e577242145e42578af6b53d80aa7d31 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7e577242145e42578af6b53d80aa7d31 2023-04-01 14:56:04,585 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e971586a3b24422cad901b3ca1422216 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e971586a3b24422cad901b3ca1422216 2023-04-01 14:56:04,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/220b7d59b99f4e7abd8c089a0adae0b9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/220b7d59b99f4e7abd8c089a0adae0b9 2023-04-01 14:56:04,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/22beee555e2d4cb8ae7be231e0113544 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/22beee555e2d4cb8ae7be231e0113544 2023-04-01 14:56:04,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/544e93c2f6a0459fa91a9f3778efcddf to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/544e93c2f6a0459fa91a9f3778efcddf 2023-04-01 14:56:04,588 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5f99100790ab4c13b44647cad6f4ecae to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5f99100790ab4c13b44647cad6f4ecae 2023-04-01 14:56:04,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/632675a61a444c72a87b598725d468d7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/632675a61a444c72a87b598725d468d7 2023-04-01 14:56:04,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3043b40fb6764642bbe3bbaac4629783 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3043b40fb6764642bbe3bbaac4629783 2023-04-01 14:56:04,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6243b27b51f54a0994a6c481735ad7c1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6243b27b51f54a0994a6c481735ad7c1 2023-04-01 14:56:04,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/b189447fd8ed40dfb8fd0873364365ab to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/b189447fd8ed40dfb8fd0873364365ab 2023-04-01 14:56:04,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/57814530ced6486093bccae7f7874f36 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/57814530ced6486093bccae7f7874f36 2023-04-01 14:56:04,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2957beb01ba543c09268d363cb15843b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2957beb01ba543c09268d363cb15843b 2023-04-01 14:56:04,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7a8c7e62321048d9997d7a0a88ad065d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/7a8c7e62321048d9997d7a0a88ad065d 2023-04-01 14:56:04,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2dc4bd2f94e34f0e932b8d8416205de4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2dc4bd2f94e34f0e932b8d8416205de4 2023-04-01 14:56:04,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/0d4d8ad2a2fb4426814e51b4faaf069c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/0d4d8ad2a2fb4426814e51b4faaf069c 2023-04-01 14:56:04,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/044d8ec440d542c9a380c5dba51a99cb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/044d8ec440d542c9a380c5dba51a99cb 2023-04-01 14:56:04,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/edbc0a52a2314d0782c5dfe5e20f514b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/edbc0a52a2314d0782c5dfe5e20f514b 2023-04-01 14:56:04,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c1d09a0d98c44ed6a201ac86819a7c47 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c1d09a0d98c44ed6a201ac86819a7c47 2023-04-01 14:56:04,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/18a6975879fb41659cacedde8fa6d674 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/18a6975879fb41659cacedde8fa6d674 2023-04-01 14:56:04,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/a32203114bd2459e84f5395741bdfe49 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/a32203114bd2459e84f5395741bdfe49 2023-04-01 14:56:04,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d55066b5e4534ca8a1d5468e4083411d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d55066b5e4534ca8a1d5468e4083411d 2023-04-01 14:56:04,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/cc057543248c4f2e95b5055a140210d0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/cc057543248c4f2e95b5055a140210d0 2023-04-01 14:56:04,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/65d59e3faff14edab5c9a2c901f8a3f7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/65d59e3faff14edab5c9a2c901f8a3f7 2023-04-01 14:56:04,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/68666cc5363b412ea262cb48487bb737 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/68666cc5363b412ea262cb48487bb737 2023-04-01 14:56:04,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/00edfd55ca76480a99658f9d320ab91a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/00edfd55ca76480a99658f9d320ab91a 2023-04-01 14:56:04,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/35e38d944ffd4095b108b865c0113a1a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/35e38d944ffd4095b108b865c0113a1a 2023-04-01 14:56:04,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/55220ae246934adbae18621d67723774 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/55220ae246934adbae18621d67723774 2023-04-01 14:56:04,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c8a33789e19b4438907fffef3cd5c9d1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c8a33789e19b4438907fffef3cd5c9d1 2023-04-01 14:56:04,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/781c9e92594c46a7a44e3c3dad80c50e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/781c9e92594c46a7a44e3c3dad80c50e 2023-04-01 14:56:04,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/f8a1b9f717ac40ad817807407a402b03 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/f8a1b9f717ac40ad817807407a402b03 2023-04-01 14:56:04,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/421794020ccc48d3ad655d62305d1192 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/421794020ccc48d3ad655d62305d1192 2023-04-01 14:56:04,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5846d9a730ed4d5a916f1e8431b77c1d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5846d9a730ed4d5a916f1e8431b77c1d 2023-04-01 14:56:04,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6b95981e596643cbafc2e6e567435fd5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6b95981e596643cbafc2e6e567435fd5 2023-04-01 14:56:04,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d0f0866690a940cbaf20f2b1d65a3732 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d0f0866690a940cbaf20f2b1d65a3732 2023-04-01 14:56:04,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/47eb033ec2b844b496ceeb692259984f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/47eb033ec2b844b496ceeb692259984f 2023-04-01 14:56:04,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c368ddf486b34008b65b9f6d54336a3f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c368ddf486b34008b65b9f6d54336a3f 2023-04-01 14:56:04,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c580be18480b417c8ff185824702215b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/c580be18480b417c8ff185824702215b 2023-04-01 14:56:04,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d6ba749048e548249025bc07d84d93eb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/d6ba749048e548249025bc07d84d93eb 2023-04-01 14:56:04,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6620d59398614225a6994f29e35a6031 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/6620d59398614225a6994f29e35a6031 2023-04-01 14:56:04,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5a003de23c0846ba8b3eb7c38db98dd4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/5a003de23c0846ba8b3eb7c38db98dd4 2023-04-01 14:56:04,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3cfdead8adbd4b05b826b6e77310406e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/3cfdead8adbd4b05b826b6e77310406e 2023-04-01 14:56:04,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/1d471440c75f43a09f37b52b0e1d6daf to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/1d471440c75f43a09f37b52b0e1d6daf 2023-04-01 14:56:04,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2a6aba8865dd4cd683f9f90fa4514007 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/2a6aba8865dd4cd683f9f90fa4514007 2023-04-01 14:56:04,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e9d75e3761a64032aab673123d558473 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/e9d75e3761a64032aab673123d558473 2023-04-01 14:56:04,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aef386d3ed944fba89ca6708467e4bed to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/aef386d3ed944fba89ca6708467e4bed 2023-04-01 14:56:04,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/86d22adc6de6473895e2ec3b71b4edf0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/86d22adc6de6473895e2ec3b71b4edf0 2023-04-01 14:56:04,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5164830874fd4eb5b09723a6f572640c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2767b0a29fce44439c4482d78cf98441, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ef2d03f682b548719f35e193f5f8d8cb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/9a50edac565f43a69dce8314b1ba9d02, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/d4256e958bb644d091fc649b540b3719, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/fd22b55cd0fd4d4a98c9464c622ec69c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ab73bab655ad4ecbbf4531972267ea59, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/275b1558f73e43e0adeeb77357903617, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1c1d1142939d473c8b4199f1ac2f085c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/207e3055252c4446a73bd52a9fcaa828, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/239b89c38ac3478b8e147e43c56ba919, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/286bab0c821b446cad2f1a3d1400aeff, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b0e674757cbf40f894f6c05f9586a4c9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/177adc8a122e4bb2bae4269614579c6e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2f9b695b4b574049b4b433ca5bb0284a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2414ea28cc5045e3a95c1b9d27fe9e7f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5348a97b3fee46d8b92f3cb7a288b6de, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2558606886204e4187525ab6cd6d3698, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ded4aaa683e6491faba333c1de387285, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7b5d598e67b64916a4dcc2bf36e1fca4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/cd4b0c019fa34a0dbcdba2218486d0b3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e9bd7cf001d2464fa90b71e7a16ea632, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8c5b94ee725e4f57a1dc6aa68e6afbb1, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/6e047409777b47738c28462cc0ca8061, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/357a256b4a3d43bca28f2688f1c0e8d7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a5df6c2338fc4f68b2021ac44e24fed5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8b26b40b60804873bd35314fa705f193, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/693213ca2376448081d7ef7cb2b8e0ed, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e22514a0910b4b7da5b1b918892855d6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5d703202d5494d2fb34488c72610bac3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7a8cd61b26fe4aaf8f6f1a228e9d0209, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b717596377c34b56802c0be06272cfc2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e188b7c8e4314d46852f7eeae8e3b52b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a71e0a7100ec4f5687d9529cc987530e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/85fcc3c43d1244aab8c2853e44acd15f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/99760ef96ea84bbcb9c7d41cdbe4042c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2121bdafd88a4715b474562be5eee12a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a94633b81cc649ed847372a0edb3cdc6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b7c83d7fe3944adc81903e6e50fb249c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a56a7367b25c4a8fad4a566ece309535, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/32697aac1d54495bb2303d2dd4c4db31, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2412accd4aa5422188a02a54af90925c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1128c06117a04071ada5405c1b78179e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7c6c5b3edac64718ad64acc9eaf23737, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2420216877df4ebe85751bb8de295643, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/0d0e6d56c7a54bbc95a990c54fd36f6c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ecb753aa3f744c6ea76a91d01f9f685b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8675329f57b6440fbe014d9e5c12d741, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e32dbdf7f07f4802b7c1a49f9b8f20a4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a8c37a76ee9540b1b1682c1c62607f02, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/3b1a9f9987e54e83b73ba4be5cf53bf4] to archive 2023-04-01 14:56:04,625 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:56:04,626 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5164830874fd4eb5b09723a6f572640c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5164830874fd4eb5b09723a6f572640c 2023-04-01 14:56:04,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2767b0a29fce44439c4482d78cf98441 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2767b0a29fce44439c4482d78cf98441 2023-04-01 14:56:04,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ef2d03f682b548719f35e193f5f8d8cb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ef2d03f682b548719f35e193f5f8d8cb 2023-04-01 14:56:04,629 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/9a50edac565f43a69dce8314b1ba9d02 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/9a50edac565f43a69dce8314b1ba9d02 2023-04-01 14:56:04,629 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/d4256e958bb644d091fc649b540b3719 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/d4256e958bb644d091fc649b540b3719 2023-04-01 14:56:04,630 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/fd22b55cd0fd4d4a98c9464c622ec69c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/fd22b55cd0fd4d4a98c9464c622ec69c 2023-04-01 14:56:04,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ab73bab655ad4ecbbf4531972267ea59 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ab73bab655ad4ecbbf4531972267ea59 2023-04-01 14:56:04,632 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/275b1558f73e43e0adeeb77357903617 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/275b1558f73e43e0adeeb77357903617 2023-04-01 14:56:04,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1c1d1142939d473c8b4199f1ac2f085c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1c1d1142939d473c8b4199f1ac2f085c 2023-04-01 14:56:04,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/207e3055252c4446a73bd52a9fcaa828 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/207e3055252c4446a73bd52a9fcaa828 2023-04-01 14:56:04,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/239b89c38ac3478b8e147e43c56ba919 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/239b89c38ac3478b8e147e43c56ba919 2023-04-01 14:56:04,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/286bab0c821b446cad2f1a3d1400aeff to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/286bab0c821b446cad2f1a3d1400aeff 2023-04-01 14:56:04,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b0e674757cbf40f894f6c05f9586a4c9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b0e674757cbf40f894f6c05f9586a4c9 2023-04-01 14:56:04,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/177adc8a122e4bb2bae4269614579c6e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/177adc8a122e4bb2bae4269614579c6e 2023-04-01 14:56:04,637 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2f9b695b4b574049b4b433ca5bb0284a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2f9b695b4b574049b4b433ca5bb0284a 2023-04-01 14:56:04,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2414ea28cc5045e3a95c1b9d27fe9e7f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2414ea28cc5045e3a95c1b9d27fe9e7f 2023-04-01 14:56:04,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5348a97b3fee46d8b92f3cb7a288b6de to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5348a97b3fee46d8b92f3cb7a288b6de 2023-04-01 14:56:04,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2558606886204e4187525ab6cd6d3698 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2558606886204e4187525ab6cd6d3698 2023-04-01 14:56:04,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ded4aaa683e6491faba333c1de387285 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ded4aaa683e6491faba333c1de387285 2023-04-01 14:56:04,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7b5d598e67b64916a4dcc2bf36e1fca4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7b5d598e67b64916a4dcc2bf36e1fca4 2023-04-01 14:56:04,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/cd4b0c019fa34a0dbcdba2218486d0b3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/cd4b0c019fa34a0dbcdba2218486d0b3 2023-04-01 14:56:04,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e9bd7cf001d2464fa90b71e7a16ea632 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e9bd7cf001d2464fa90b71e7a16ea632 2023-04-01 14:56:04,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8c5b94ee725e4f57a1dc6aa68e6afbb1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8c5b94ee725e4f57a1dc6aa68e6afbb1 2023-04-01 14:56:04,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/6e047409777b47738c28462cc0ca8061 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/6e047409777b47738c28462cc0ca8061 2023-04-01 14:56:04,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/357a256b4a3d43bca28f2688f1c0e8d7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/357a256b4a3d43bca28f2688f1c0e8d7 2023-04-01 14:56:04,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a5df6c2338fc4f68b2021ac44e24fed5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a5df6c2338fc4f68b2021ac44e24fed5 2023-04-01 14:56:04,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8b26b40b60804873bd35314fa705f193 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8b26b40b60804873bd35314fa705f193 2023-04-01 14:56:04,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/693213ca2376448081d7ef7cb2b8e0ed to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/693213ca2376448081d7ef7cb2b8e0ed 2023-04-01 14:56:04,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e22514a0910b4b7da5b1b918892855d6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e22514a0910b4b7da5b1b918892855d6 2023-04-01 14:56:04,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5d703202d5494d2fb34488c72610bac3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/5d703202d5494d2fb34488c72610bac3 2023-04-01 14:56:04,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7a8cd61b26fe4aaf8f6f1a228e9d0209 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7a8cd61b26fe4aaf8f6f1a228e9d0209 2023-04-01 14:56:04,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b717596377c34b56802c0be06272cfc2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b717596377c34b56802c0be06272cfc2 2023-04-01 14:56:04,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e188b7c8e4314d46852f7eeae8e3b52b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e188b7c8e4314d46852f7eeae8e3b52b 2023-04-01 14:56:04,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a71e0a7100ec4f5687d9529cc987530e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a71e0a7100ec4f5687d9529cc987530e 2023-04-01 14:56:04,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/85fcc3c43d1244aab8c2853e44acd15f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/85fcc3c43d1244aab8c2853e44acd15f 2023-04-01 14:56:04,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/99760ef96ea84bbcb9c7d41cdbe4042c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/99760ef96ea84bbcb9c7d41cdbe4042c 2023-04-01 14:56:04,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2121bdafd88a4715b474562be5eee12a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2121bdafd88a4715b474562be5eee12a 2023-04-01 14:56:04,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a94633b81cc649ed847372a0edb3cdc6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a94633b81cc649ed847372a0edb3cdc6 2023-04-01 14:56:04,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b7c83d7fe3944adc81903e6e50fb249c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/b7c83d7fe3944adc81903e6e50fb249c 2023-04-01 14:56:04,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a56a7367b25c4a8fad4a566ece309535 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a56a7367b25c4a8fad4a566ece309535 2023-04-01 14:56:04,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/32697aac1d54495bb2303d2dd4c4db31 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/32697aac1d54495bb2303d2dd4c4db31 2023-04-01 14:56:04,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2412accd4aa5422188a02a54af90925c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2412accd4aa5422188a02a54af90925c 2023-04-01 14:56:04,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1128c06117a04071ada5405c1b78179e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/1128c06117a04071ada5405c1b78179e 2023-04-01 14:56:04,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7c6c5b3edac64718ad64acc9eaf23737 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/7c6c5b3edac64718ad64acc9eaf23737 2023-04-01 14:56:04,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2420216877df4ebe85751bb8de295643 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/2420216877df4ebe85751bb8de295643 2023-04-01 14:56:04,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/0d0e6d56c7a54bbc95a990c54fd36f6c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/0d0e6d56c7a54bbc95a990c54fd36f6c 2023-04-01 14:56:04,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ecb753aa3f744c6ea76a91d01f9f685b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/ecb753aa3f744c6ea76a91d01f9f685b 2023-04-01 14:56:04,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8675329f57b6440fbe014d9e5c12d741 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/8675329f57b6440fbe014d9e5c12d741 2023-04-01 14:56:04,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e32dbdf7f07f4802b7c1a49f9b8f20a4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/e32dbdf7f07f4802b7c1a49f9b8f20a4 2023-04-01 14:56:04,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a8c37a76ee9540b1b1682c1c62607f02 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/a8c37a76ee9540b1b1682c1c62607f02 2023-04-01 14:56:04,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/3b1a9f9987e54e83b73ba4be5cf53bf4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/3b1a9f9987e54e83b73ba4be5cf53bf4 2023-04-01 14:56:04,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/143d884e4a3e419ab77f90b5e14010ff, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/28542dd054224b80b465225eb60f205c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/6a8131293d624cf0b17a39fc81c8cdce, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4e8f253466aa444eba76dd3739794a49, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/45b45f0fe50547d29a84f0a9de263fde, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/c836d2954e494f8fac3da7ea6b0d88c0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bfc4ae97628f478aa76302a54b6a4d3e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8087e2f8d0164b15b774c1684d0c1a78, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf813067646a441883087a590fa11ebf, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8524f835b0184788856844116815c4a4, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ccd61d6b54f64e4997b1b4c97c8bf290, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/58b4b7c0a0b944c49c17bd82f5e61bf6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7e097be164d74f1aa71294ec29a3c316, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/cfdafa2650484cdebd8bbae2c84a9a48, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b91ecc87b3444541af0f3422c832eb62, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/86b4bd188a7f40f6b30691ed3d1fad8a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/18fb72ac835644aebf0e158434eeeb59, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f346e12e5fa84d31a1939a06eac7053f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/23fbfc9d9acf4bf3a78b8da7f1a39533, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/34f66e69bfd54f2985477649529566b0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8068d87861f64b7a8abf9f3af605e874, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/0ac8c71699d24b378347db046d1f2905, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f3b19c7e7e544b3591216c58f8cdfc63, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf6dbcf2668a4fc7988be93b606a9e25, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/fb2b3379a55a44d598c647ae12f12492, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/da73d16c721c475fb1d4d9a6b474ea92, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ea30ee8fb47148b381f66bb4b93758bb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b617ac809a874305b218d917726fe730, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/da4c4ec06f664f0cbec5a729f6e39c10, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ece89fb609e14a27bafc0ec997dee17a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/a995aab5493e459b8ac18c9b5b2bd6af, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/14ae1682c53146e69901bc03a9fdcec8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/79b115cc0e7f4781a17c8ae9551c518d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/9f0a20b02beb4adcbb160d0a9637ca8c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b6cebcf535b246808cb9e87c780e550a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7ab321cee76f469bab1907b60208900d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/05819d7750174df2ac2c9a3d25a12ab8, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f512032209a34a22b9e3a98e1682afd3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/3ddeac8365fb45448c3f251e7789b3c0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/0ef8c263721a4867927e89a951b5171a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4b84357ac8d84b83b51fd3b4d0aeb879, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7110aa52983a41f382b75bdad264b10f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/854c39466248448d8269e14fad11a13c] to archive 2023-04-01 14:56:04,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:56:04,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/143d884e4a3e419ab77f90b5e14010ff to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/143d884e4a3e419ab77f90b5e14010ff 2023-04-01 14:56:04,673 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/28542dd054224b80b465225eb60f205c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/28542dd054224b80b465225eb60f205c 2023-04-01 14:56:04,673 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/6a8131293d624cf0b17a39fc81c8cdce to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/6a8131293d624cf0b17a39fc81c8cdce 2023-04-01 14:56:04,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4e8f253466aa444eba76dd3739794a49 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4e8f253466aa444eba76dd3739794a49 2023-04-01 14:56:04,675 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/45b45f0fe50547d29a84f0a9de263fde to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/45b45f0fe50547d29a84f0a9de263fde 2023-04-01 14:56:04,676 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/c836d2954e494f8fac3da7ea6b0d88c0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/c836d2954e494f8fac3da7ea6b0d88c0 2023-04-01 14:56:04,677 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bfc4ae97628f478aa76302a54b6a4d3e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bfc4ae97628f478aa76302a54b6a4d3e 2023-04-01 14:56:04,678 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8087e2f8d0164b15b774c1684d0c1a78 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8087e2f8d0164b15b774c1684d0c1a78 2023-04-01 14:56:04,679 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf813067646a441883087a590fa11ebf to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf813067646a441883087a590fa11ebf 2023-04-01 14:56:04,679 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8524f835b0184788856844116815c4a4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8524f835b0184788856844116815c4a4 2023-04-01 14:56:04,680 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ccd61d6b54f64e4997b1b4c97c8bf290 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ccd61d6b54f64e4997b1b4c97c8bf290 2023-04-01 14:56:04,681 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/58b4b7c0a0b944c49c17bd82f5e61bf6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/58b4b7c0a0b944c49c17bd82f5e61bf6 2023-04-01 14:56:04,682 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7e097be164d74f1aa71294ec29a3c316 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7e097be164d74f1aa71294ec29a3c316 2023-04-01 14:56:04,683 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/cfdafa2650484cdebd8bbae2c84a9a48 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/cfdafa2650484cdebd8bbae2c84a9a48 2023-04-01 14:56:04,684 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b91ecc87b3444541af0f3422c832eb62 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b91ecc87b3444541af0f3422c832eb62 2023-04-01 14:56:04,685 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/86b4bd188a7f40f6b30691ed3d1fad8a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/86b4bd188a7f40f6b30691ed3d1fad8a 2023-04-01 14:56:04,686 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/18fb72ac835644aebf0e158434eeeb59 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/18fb72ac835644aebf0e158434eeeb59 2023-04-01 14:56:04,687 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f346e12e5fa84d31a1939a06eac7053f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f346e12e5fa84d31a1939a06eac7053f 2023-04-01 14:56:04,687 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/23fbfc9d9acf4bf3a78b8da7f1a39533 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/23fbfc9d9acf4bf3a78b8da7f1a39533 2023-04-01 14:56:04,689 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/34f66e69bfd54f2985477649529566b0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/34f66e69bfd54f2985477649529566b0 2023-04-01 14:56:04,690 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8068d87861f64b7a8abf9f3af605e874 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/8068d87861f64b7a8abf9f3af605e874 2023-04-01 14:56:04,690 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/0ac8c71699d24b378347db046d1f2905 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/0ac8c71699d24b378347db046d1f2905 2023-04-01 14:56:04,691 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f3b19c7e7e544b3591216c58f8cdfc63 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f3b19c7e7e544b3591216c58f8cdfc63 2023-04-01 14:56:04,693 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf6dbcf2668a4fc7988be93b606a9e25 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/bf6dbcf2668a4fc7988be93b606a9e25 2023-04-01 14:56:04,694 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/fb2b3379a55a44d598c647ae12f12492 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/fb2b3379a55a44d598c647ae12f12492 2023-04-01 14:56:04,695 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/da73d16c721c475fb1d4d9a6b474ea92 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/da73d16c721c475fb1d4d9a6b474ea92 2023-04-01 14:56:04,695 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ea30ee8fb47148b381f66bb4b93758bb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ea30ee8fb47148b381f66bb4b93758bb 2023-04-01 14:56:04,696 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b617ac809a874305b218d917726fe730 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b617ac809a874305b218d917726fe730 2023-04-01 14:56:04,697 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/da4c4ec06f664f0cbec5a729f6e39c10 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/da4c4ec06f664f0cbec5a729f6e39c10 2023-04-01 14:56:04,698 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ece89fb609e14a27bafc0ec997dee17a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/ece89fb609e14a27bafc0ec997dee17a 2023-04-01 14:56:04,699 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/a995aab5493e459b8ac18c9b5b2bd6af to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/a995aab5493e459b8ac18c9b5b2bd6af 2023-04-01 14:56:04,700 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/14ae1682c53146e69901bc03a9fdcec8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/14ae1682c53146e69901bc03a9fdcec8 2023-04-01 14:56:04,701 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/79b115cc0e7f4781a17c8ae9551c518d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/79b115cc0e7f4781a17c8ae9551c518d 2023-04-01 14:56:04,701 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/9f0a20b02beb4adcbb160d0a9637ca8c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/9f0a20b02beb4adcbb160d0a9637ca8c 2023-04-01 14:56:04,702 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b6cebcf535b246808cb9e87c780e550a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/b6cebcf535b246808cb9e87c780e550a 2023-04-01 14:56:04,703 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7ab321cee76f469bab1907b60208900d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7ab321cee76f469bab1907b60208900d 2023-04-01 14:56:04,704 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/05819d7750174df2ac2c9a3d25a12ab8 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/05819d7750174df2ac2c9a3d25a12ab8 2023-04-01 14:56:04,704 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f512032209a34a22b9e3a98e1682afd3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/f512032209a34a22b9e3a98e1682afd3 2023-04-01 14:56:04,705 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/3ddeac8365fb45448c3f251e7789b3c0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/3ddeac8365fb45448c3f251e7789b3c0 2023-04-01 14:56:04,706 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/0ef8c263721a4867927e89a951b5171a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/0ef8c263721a4867927e89a951b5171a 2023-04-01 14:56:04,707 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4b84357ac8d84b83b51fd3b4d0aeb879 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/4b84357ac8d84b83b51fd3b4d0aeb879 2023-04-01 14:56:04,707 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7110aa52983a41f382b75bdad264b10f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7110aa52983a41f382b75bdad264b10f 2023-04-01 14:56:04,708 DEBUG [StoreCloser-TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/854c39466248448d8269e14fad11a13c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/854c39466248448d8269e14fad11a13c 2023-04-01 14:56:04,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/recovered.edits/744.seqid, newMaxSeqId=744, maxSeqId=1 2023-04-01 14:56:04,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15. 2023-04-01 14:56:04,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2ddd6fcff99c4aa857940e9dc8e40f15: 2023-04-01 14:56:04,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:56:04,715 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2ddd6fcff99c4aa857940e9dc8e40f15, regionState=CLOSED 2023-04-01 14:56:04,715 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360964715"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360964715"}]},"ts":"1680360964715"} 2023-04-01 14:56:04,717 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-01 14:56:04,717 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 2ddd6fcff99c4aa857940e9dc8e40f15, server=jenkins-hbase4.apache.org,40863,1680360818567 in 1.3770 sec 2023-04-01 14:56:04,718 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-01 14:56:04,718 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ddd6fcff99c4aa857940e9dc8e40f15, UNASSIGN in 1.3810 sec 2023-04-01 14:56:04,719 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360964719"}]},"ts":"1680360964719"} 2023-04-01 14:56:04,719 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 14:56:04,722 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 14:56:04,723 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.3930 sec 2023-04-01 14:56:05,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-01 14:56:05,437 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-01 14:56:05,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-01 14:56:05,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:05,440 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:05,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-01 14:56:05,441 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:05,445 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:56:05,447 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/recovered.edits] 2023-04-01 14:56:05,449 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/04a5f812e58043328bac2671a5324f14 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/04a5f812e58043328bac2671a5324f14 2023-04-01 14:56:05,450 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/422ea860e0f8477894b285bc8b3624cd to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/422ea860e0f8477894b285bc8b3624cd 2023-04-01 14:56:05,451 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/4684f28f81024ca58a84f3cb39959aa1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/A/4684f28f81024ca58a84f3cb39959aa1 2023-04-01 14:56:05,453 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/acb203d1296941a0bef94c8a372a8511 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/acb203d1296941a0bef94c8a372a8511 2023-04-01 14:56:05,454 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/bc3162184a1d4a62b8ba99dcebaeb30b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/bc3162184a1d4a62b8ba99dcebaeb30b 2023-04-01 14:56:05,455 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/c1f874c5d15547faaf6bdbc9da305dcb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/B/c1f874c5d15547faaf6bdbc9da305dcb 2023-04-01 14:56:05,457 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/00cb530961374a47afe3611e6e1eee74 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/00cb530961374a47afe3611e6e1eee74 2023-04-01 14:56:05,458 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/06a5f3f2d1094df994dd9454b488b9aa to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/06a5f3f2d1094df994dd9454b488b9aa 2023-04-01 14:56:05,459 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/3bb990bdc4254dcf83136ce0d9b5141f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/3bb990bdc4254dcf83136ce0d9b5141f 2023-04-01 14:56:05,460 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/514a61ce7f8c4c1dbc86401fc94aa9f4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/514a61ce7f8c4c1dbc86401fc94aa9f4 2023-04-01 14:56:05,461 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/59ba248de4154ed58a030e72b73c3633 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/59ba248de4154ed58a030e72b73c3633 2023-04-01 14:56:05,462 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/5d56b15dc28e409f991e639c71d3e7b1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/5d56b15dc28e409f991e639c71d3e7b1 2023-04-01 14:56:05,463 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7c11070ae57d47e6b478b9388359ea61 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/7c11070ae57d47e6b478b9388359ea61 2023-04-01 14:56:05,463 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/91eb5660697840609cb8983ed3c6ba13 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/C/91eb5660697840609cb8983ed3c6ba13 2023-04-01 14:56:05,465 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/recovered.edits/744.seqid to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15/recovered.edits/744.seqid 2023-04-01 14:56:05,466 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/2ddd6fcff99c4aa857940e9dc8e40f15 2023-04-01 14:56:05,466 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:56:05,468 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:05,469 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 14:56:05,470 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 14:56:05,471 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:05,471 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 14:56:05,471 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680360965471"}]},"ts":"9223372036854775807"} 2023-04-01 14:56:05,472 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 14:56:05,472 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2ddd6fcff99c4aa857940e9dc8e40f15, NAME => 'TestAcidGuarantees,,1680360940905.2ddd6fcff99c4aa857940e9dc8e40f15.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 14:56:05,472 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 14:56:05,472 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680360965472"}]},"ts":"9223372036854775807"} 2023-04-01 14:56:05,473 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 14:56:05,475 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:05,476 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 37 msec 2023-04-01 14:56:05,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-01 14:56:05,542 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-01 14:56:05,553 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=234 (was 229) - Thread LEAK? -, OpenFileDescriptor=618 (was 524) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=413 (was 360) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=3301 (was 3139) - AvailableMemoryMB LEAK? - 2023-04-01 14:56:05,566 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=234, OpenFileDescriptor=618, MaxFileDescriptor=60000, SystemLoadAverage=413, ProcessCount=172, AvailableMemoryMB=3301 2023-04-01 14:56:05,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 14:56:05,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 14:56:05,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:05,570 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-01 14:56:05,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-01 14:56:05,570 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-01 14:56:05,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-01 14:56:05,571 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:05,572 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4 empty. 2023-04-01 14:56:05,572 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:05,572 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:56:05,578 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 14:56:05,579 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d5537299121eaceb1e94c26142bdabf4, NAME => 'TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp 2023-04-01 14:56:05,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-01 14:56:05,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-01 14:56:05,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:56:05,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d5537299121eaceb1e94c26142bdabf4, disabling compactions & flushes 2023-04-01 14:56:05,984 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:05,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:05,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. after waiting 0 ms 2023-04-01 14:56:05,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:05,984 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:05,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:05,986 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-01 14:56:05,987 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360965986"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360965986"}]},"ts":"1680360965986"} 2023-04-01 14:56:05,988 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-01 14:56:05,988 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-01 14:56:05,988 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360965988"}]},"ts":"1680360965988"} 2023-04-01 14:56:05,989 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-01 14:56:05,992 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d5537299121eaceb1e94c26142bdabf4, ASSIGN}] 2023-04-01 14:56:05,993 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d5537299121eaceb1e94c26142bdabf4, ASSIGN 2023-04-01 14:56:05,994 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d5537299121eaceb1e94c26142bdabf4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40863,1680360818567; forceNewPlan=false, retain=false 2023-04-01 14:56:06,145 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=d5537299121eaceb1e94c26142bdabf4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:06,145 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360966145"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360966145"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360966145"}]},"ts":"1680360966145"} 2023-04-01 14:56:06,146 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:56:06,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-01 14:56:06,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:06,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d5537299121eaceb1e94c26142bdabf4, NAME => 'TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:56:06,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:06,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:56:06,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:06,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:06,302 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:06,303 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A 2023-04-01 14:56:06,303 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A 2023-04-01 14:56:06,303 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:56:06,303 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d5537299121eaceb1e94c26142bdabf4 columnFamilyName A 2023-04-01 14:56:06,304 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(310): Store=d5537299121eaceb1e94c26142bdabf4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:56:06,304 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:06,305 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B 2023-04-01 14:56:06,305 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B 2023-04-01 14:56:06,305 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:56:06,305 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d5537299121eaceb1e94c26142bdabf4 columnFamilyName B 2023-04-01 14:56:06,306 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(310): Store=d5537299121eaceb1e94c26142bdabf4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:56:06,306 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:06,307 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C 2023-04-01 14:56:06,307 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C 2023-04-01 14:56:06,307 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:56:06,307 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d5537299121eaceb1e94c26142bdabf4 columnFamilyName C 2023-04-01 14:56:06,308 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(310): Store=d5537299121eaceb1e94c26142bdabf4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:56:06,308 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:06,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:06,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:06,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 14:56:06,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:06,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-01 14:56:06,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d5537299121eaceb1e94c26142bdabf4; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62542386, jitterRate=-0.06804582476615906}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 14:56:06,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:06,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., pid=56, masterSystemTime=1680360966297 2023-04-01 14:56:06,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:06,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:06,314 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=d5537299121eaceb1e94c26142bdabf4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:06,314 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360966314"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680360966314"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680360966314"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680360966314"}]},"ts":"1680360966314"} 2023-04-01 14:56:06,316 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-01 14:56:06,316 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 in 169 msec 2023-04-01 14:56:06,318 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-01 14:56:06,318 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d5537299121eaceb1e94c26142bdabf4, ASSIGN in 324 msec 2023-04-01 14:56:06,318 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-01 14:56:06,318 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360966318"}]},"ts":"1680360966318"} 2023-04-01 14:56:06,319 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-01 14:56:06,322 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-01 14:56:06,323 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 754 msec 2023-04-01 14:56:06,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-01 14:56:06,674 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-01 14:56:06,675 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:56:06,680 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:56:06,681 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:56:06,682 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41728, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:56:06,683 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-01 14:56:06,685 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34590, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-01 14:56:06,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-01 14:56:06,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-01 14:56:06,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:07,104 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 14:56:07,105 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-01 14:56:07,105 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-01 14:56:07,107 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-01 14:56:07,109 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d5537299121eaceb1e94c26142bdabf4, REOPEN/MOVE}] 2023-04-01 14:56:07,109 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d5537299121eaceb1e94c26142bdabf4, REOPEN/MOVE 2023-04-01 14:56:07,110 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d5537299121eaceb1e94c26142bdabf4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,110 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360967110"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360967110"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360967110"}]},"ts":"1680360967110"} 2023-04-01 14:56:07,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:56:07,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d5537299121eaceb1e94c26142bdabf4, disabling compactions & flushes 2023-04-01 14:56:07,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. after waiting 0 ms 2023-04-01 14:56:07,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-01 14:56:07,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:07,267 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: d5537299121eaceb1e94c26142bdabf4 to self. 2023-04-01 14:56:07,268 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,269 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d5537299121eaceb1e94c26142bdabf4, regionState=CLOSED 2023-04-01 14:56:07,269 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360967269"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360967269"}]},"ts":"1680360967269"} 2023-04-01 14:56:07,271 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-01 14:56:07,271 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 in 159 msec 2023-04-01 14:56:07,271 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d5537299121eaceb1e94c26142bdabf4, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40863,1680360818567; forceNewPlan=false, retain=true 2023-04-01 14:56:07,422 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d5537299121eaceb1e94c26142bdabf4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,422 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360967421"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360967421"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360967421"}]},"ts":"1680360967421"} 2023-04-01 14:56:07,423 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:56:07,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d5537299121eaceb1e94c26142bdabf4, NAME => 'TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.', STARTKEY => '', ENDKEY => ''} 2023-04-01 14:56:07,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-01 14:56:07,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,579 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,580 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A 2023-04-01 14:56:07,580 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A 2023-04-01 14:56:07,580 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:56:07,580 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d5537299121eaceb1e94c26142bdabf4 columnFamilyName A 2023-04-01 14:56:07,581 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(310): Store=d5537299121eaceb1e94c26142bdabf4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:56:07,582 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,582 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B 2023-04-01 14:56:07,582 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B 2023-04-01 14:56:07,582 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:56:07,582 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d5537299121eaceb1e94c26142bdabf4 columnFamilyName B 2023-04-01 14:56:07,583 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(310): Store=d5537299121eaceb1e94c26142bdabf4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:56:07,583 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,583 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C 2023-04-01 14:56:07,584 DEBUG [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C 2023-04-01 14:56:07,584 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-01 14:56:07,584 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d5537299121eaceb1e94c26142bdabf4 columnFamilyName C 2023-04-01 14:56:07,584 INFO [StoreOpener-d5537299121eaceb1e94c26142bdabf4-1] regionserver.HStore(310): Store=d5537299121eaceb1e94c26142bdabf4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-01 14:56:07,584 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-01 14:56:07,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d5537299121eaceb1e94c26142bdabf4; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65748069, jitterRate=-0.02027742564678192}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-01 14:56:07,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:07,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., pid=61, masterSystemTime=1680360967574 2023-04-01 14:56:07,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,590 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,591 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d5537299121eaceb1e94c26142bdabf4, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,591 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360967591"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680360967591"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680360967591"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680360967591"}]},"ts":"1680360967591"} 2023-04-01 14:56:07,593 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-01 14:56:07,593 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 in 169 msec 2023-04-01 14:56:07,594 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-01 14:56:07,594 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d5537299121eaceb1e94c26142bdabf4, REOPEN/MOVE in 484 msec 2023-04-01 14:56:07,595 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-01 14:56:07,595 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 487 msec 2023-04-01 14:56:07,596 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 908 msec 2023-04-01 14:56:07,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-01 14:56:07,598 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:56:07,602 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:56:07,603 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:56:07,608 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:56:07,608 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:56:07,615 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:56:07,616 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:56:07,619 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:56:07,619 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x5e6a6f3c to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:56:07,623 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f4eab6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:56:07,623 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:56:07,630 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:56:07,630 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:56:07,637 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:56:07,638 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x4fae3ed2 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:56:07,644 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28164190, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:56:07,644 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x0eedd898 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:56:07,650 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@308b3f39, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:56:07,650 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x618c0022 to 127.0.0.1:60465 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-01 14:56:07,659 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1453d93, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-01 14:56:07,672 DEBUG [hconnection-0x5cabf54f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:56:07,676 DEBUG [hconnection-0x7c6e3c9d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:56:07,676 DEBUG [hconnection-0x31d0f30e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:56:07,676 DEBUG [hconnection-0x50bf2e8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:56:07,676 DEBUG [hconnection-0x53f7fd54-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:56:07,677 DEBUG [hconnection-0x60b667be-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:56:07,677 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:56:07,679 DEBUG [hconnection-0x6ee678f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:56:07,679 DEBUG [hconnection-0x6b9368bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:56:07,680 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:56:07,691 DEBUG [hconnection-0x90f9bd1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:56:07,691 DEBUG [hconnection-0x64634d16-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-01 14:56:07,691 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41736, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:56:07,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:07,692 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41772, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:56:07,692 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:56:07,693 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41794, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:56:07,695 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41734, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:56:07,695 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41762, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:56:07,695 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:56:07,695 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-01 14:56:07,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:07,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:07,700 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:07,700 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:07,702 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:07,702 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:07,703 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,703 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,706 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:07,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:07,707 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,707 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,707 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:07,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:07,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-01 14:56:07,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:07,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:07,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:07,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:07,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:07,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:07,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:07,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:07,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:07,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:07,712 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:07,716 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:07,716 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:07,716 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,716 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:07,716 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:07,716 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,716 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:07,717 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:07,717 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:07,724 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:07,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361027722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361027723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361027725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,728 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,728 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:07,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:07,728 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,728 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:07,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:07,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361027727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:07,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361027726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:07,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:07,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f02d573[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:07,730 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:07,730 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,731 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,731 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:07,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,732 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:07,732 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:07,732 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:07,733 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:07,733 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:07,733 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:07,733 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:07,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:07,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:07,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:07,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:07,742 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:07,742 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:07,742 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:07,742 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:07,742 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:07,746 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,746 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,746 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:07,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:07,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,749 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,752 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:07,752 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:07,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:07,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,752 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:07,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:07,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:07,753 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:07,753 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:07,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:07,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:07,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:07,759 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401b241d29496be4c7d9fa255a0bb212887 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b241d29496be4c7d9fa255a0bb212887 2023-04-01 14:56:07,773 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/5ccdc27eb0704b908d36e610535e8885 2023-04-01 14:56:07,794 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/da1f89fbc94b4280877a99ef5e210ea3 2023-04-01 14:56:07,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/44bcad13ffe24e46bd7cc6e5f71aaa0b 2023-04-01 14:56:07,813 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/5ccdc27eb0704b908d36e610535e8885 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5ccdc27eb0704b908d36e610535e8885 2023-04-01 14:56:07,816 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5ccdc27eb0704b908d36e610535e8885, entries=150, sequenceid=17, filesize=25.1 K 2023-04-01 14:56:07,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/da1f89fbc94b4280877a99ef5e210ea3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/da1f89fbc94b4280877a99ef5e210ea3 2023-04-01 14:56:07,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/da1f89fbc94b4280877a99ef5e210ea3, entries=150, sequenceid=17, filesize=11.6 K 2023-04-01 14:56:07,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/44bcad13ffe24e46bd7cc6e5f71aaa0b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44bcad13ffe24e46bd7cc6e5f71aaa0b 2023-04-01 14:56:07,826 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44bcad13ffe24e46bd7cc6e5f71aaa0b, entries=150, sequenceid=17, filesize=11.6 K 2023-04-01 14:56:07,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d5537299121eaceb1e94c26142bdabf4 in 116ms, sequenceid=17, compaction requested=false 2023-04-01 14:56:07,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:07,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,828 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 14:56:07,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:07,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:07,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:07,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:07,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:07,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:07,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361027835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361027836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361027836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361027839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361027839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,853 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401b6ca74388fb941d39bd2903f66efb926 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b6ca74388fb941d39bd2903f66efb926 2023-04-01 14:56:07,853 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:07,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:07,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:07,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:07,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:07,860 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:07,860 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:07,860 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=38, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/d7c8e4d1229d4695a23bacf095693038 2023-04-01 14:56:07,860 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:07,860 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:07,861 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,861 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,862 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:07,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:07,862 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,863 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,863 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:07,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:07,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:07,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:07,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:07,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:07,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:07,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:07,865 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,865 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:07,865 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:07,865 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:07,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:07,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:07,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:07,865 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:07,867 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:07,867 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,867 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:07,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:07,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:07,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:07,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:07,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@258ef8a7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:07,870 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:07,870 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,872 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:07,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:07,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:07,872 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,872 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:07,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/d87eb911a1644f2abadcd9f58247578a 2023-04-01 14:56:07,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:07,874 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,875 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:07,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:07,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:07,875 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:07,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:07,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:07,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:07,877 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:07,878 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:07,878 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:07,878 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:07,878 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:07,880 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:07,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:07,880 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,880 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:07,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:07,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:07,881 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,883 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:07,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,886 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:07,886 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:07,886 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:07,886 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:07,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:07,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:07,886 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,887 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:07,887 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:07,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:07,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:07,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/db8586053c2f4a9199f70d9b9e4dda7d 2023-04-01 14:56:07,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/d7c8e4d1229d4695a23bacf095693038 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d7c8e4d1229d4695a23bacf095693038 2023-04-01 14:56:07,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d7c8e4d1229d4695a23bacf095693038, entries=150, sequenceid=38, filesize=25.1 K 2023-04-01 14:56:07,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/d87eb911a1644f2abadcd9f58247578a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d87eb911a1644f2abadcd9f58247578a 2023-04-01 14:56:07,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d87eb911a1644f2abadcd9f58247578a, entries=150, sequenceid=38, filesize=11.6 K 2023-04-01 14:56:07,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/db8586053c2f4a9199f70d9b9e4dda7d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/db8586053c2f4a9199f70d9b9e4dda7d 2023-04-01 14:56:07,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/db8586053c2f4a9199f70d9b9e4dda7d, entries=150, sequenceid=38, filesize=11.6 K 2023-04-01 14:56:07,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for d5537299121eaceb1e94c26142bdabf4 in 78ms, sequenceid=38, compaction requested=false 2023-04-01 14:56:07,906 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:07,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:07,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-01 14:56:07,943 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:07,943 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:07,943 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:07,943 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:07,943 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:07,943 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:07,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361027958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361027958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,969 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401c0a0e3a98aae4f809a0eea41c1b25f14 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c0a0e3a98aae4f809a0eea41c1b25f14 2023-04-01 14:56:07,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361027965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361027965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:07,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361027967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,987 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:07,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:07,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:07,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:07,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:07,990 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:07,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:07,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:07,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:07,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,993 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:07,993 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:07,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:07,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:07,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:07,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:07,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:07,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:07,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:07,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:07,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:07,997 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:07,997 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:07,997 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:08,003 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:08,003 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,004 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:08,004 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:08,004 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,003 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:08,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:08,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae8721f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:08,011 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:08,011 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,012 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:08,012 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,012 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:08,013 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:08,016 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:08,016 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:08,016 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:08,016 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:08,016 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:08,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:08,019 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,020 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,023 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,023 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,023 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,023 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,024 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:08,024 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:08,024 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,024 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,024 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:08,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:08,024 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:08,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:08,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:08,024 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:08,024 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:08,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:08,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361028068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361028068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361028074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361028074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361028076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,124 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:08,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:08,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:08,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:08,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:08,128 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:08,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:08,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:08,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:08,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,128 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,130 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:08,130 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:08,130 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,130 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:08,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:08,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:08,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:08,131 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:08,131 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:08,131 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:08,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:08,132 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:08,132 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,133 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:08,133 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:08,133 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:08,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:08,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:08,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:08,134 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,134 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,135 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,135 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:08,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:08,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763a2b05[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:08,136 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:08,137 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,138 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:08,138 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,138 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:08,141 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,141 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:08,141 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,141 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:08,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:08,143 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:08,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:08,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:08,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:08,143 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:08,145 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,145 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,145 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:08,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,147 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,153 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:08,155 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:08,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:08,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:08,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:08,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:08,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:08,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:08,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:08,155 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:08,156 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:08,256 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:08,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:08,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:08,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:08,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:08,259 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:08,260 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:08,260 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:08,260 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:08,260 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,260 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,261 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:08,262 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:08,262 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,262 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:08,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:08,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:08,263 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:08,263 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:08,264 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:08,264 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:08,264 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:08,264 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,264 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:08,264 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:08,264 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,264 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:08,264 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:08,264 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,264 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:08,264 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:08,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:08,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:08,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f6ad1fc[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:08,269 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:08,269 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,270 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,270 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:08,270 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:08,270 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:08,270 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,271 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:08,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361028270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361028271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,272 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,272 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:08,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,272 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:08,272 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:08,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:08,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:08,275 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:08,276 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:08,276 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:08,276 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:08,276 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:08,277 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,277 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:08,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,278 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:08,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:08,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361028277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,279 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361028278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,280 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361028279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,284 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:08,284 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:08,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:08,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:08,284 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,284 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,284 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:08,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:08,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:08,284 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,284 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:08,284 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:08,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:08,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:08,284 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,385 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:08,385 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=53, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/e9bd859fff7b4ac99ec01c86d28c48bf 2023-04-01 14:56:08,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:08,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:08,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:08,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:08,388 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:08,388 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:08,388 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:08,388 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:08,389 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,389 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,390 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:08,390 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:08,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:08,391 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,391 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:08,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:08,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:08,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:08,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:08,392 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:08,392 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:08,392 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:08,392 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:08,392 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:08,393 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,393 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:08,393 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:08,393 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,393 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:08,393 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:08,393 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,393 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:08,393 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:08,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,395 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,395 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,395 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:08,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:08,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d735b76[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:08,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:08,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,400 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:08,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:08,400 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:08,400 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,400 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:08,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,406 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:08,406 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:08,406 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,407 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/45db72dc0c814017b5756292cf0bed9d 2023-04-01 14:56:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:08,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:08,410 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:08,410 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:08,410 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:08,410 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:08,410 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:08,413 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,413 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:08,413 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,413 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:08,418 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:08,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:08,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:08,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:08,422 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:08,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:08,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:08,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:08,422 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:08,422 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:08,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:08,422 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:08,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:08,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:08,433 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/938ddb9e82c64e9ea36f78bd90d4d7ca 2023-04-01 14:56:08,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/e9bd859fff7b4ac99ec01c86d28c48bf as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e9bd859fff7b4ac99ec01c86d28c48bf 2023-04-01 14:56:08,442 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e9bd859fff7b4ac99ec01c86d28c48bf, entries=250, sequenceid=53, filesize=38.7 K 2023-04-01 14:56:08,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/45db72dc0c814017b5756292cf0bed9d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/45db72dc0c814017b5756292cf0bed9d 2023-04-01 14:56:08,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/45db72dc0c814017b5756292cf0bed9d, entries=150, sequenceid=53, filesize=11.6 K 2023-04-01 14:56:08,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/938ddb9e82c64e9ea36f78bd90d4d7ca as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/938ddb9e82c64e9ea36f78bd90d4d7ca 2023-04-01 14:56:08,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/938ddb9e82c64e9ea36f78bd90d4d7ca, entries=150, sequenceid=53, filesize=11.6 K 2023-04-01 14:56:08,453 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d5537299121eaceb1e94c26142bdabf4 in 511ms, sequenceid=53, compaction requested=true 2023-04-01 14:56:08,453 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:08,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:08,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:08,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:08,453 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:56:08,453 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:56:08,454 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:56:08,454 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:56:08,454 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/B is initiating minor compaction (all files) 2023-04-01 14:56:08,454 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/A is initiating minor compaction (all files) 2023-04-01 14:56:08,454 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/B in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,454 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/A in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,454 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/da1f89fbc94b4280877a99ef5e210ea3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d87eb911a1644f2abadcd9f58247578a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/45db72dc0c814017b5756292cf0bed9d] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=34.8 K 2023-04-01 14:56:08,454 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5ccdc27eb0704b908d36e610535e8885, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d7c8e4d1229d4695a23bacf095693038, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e9bd859fff7b4ac99ec01c86d28c48bf] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=89.0 K 2023-04-01 14:56:08,455 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting da1f89fbc94b4280877a99ef5e210ea3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680360967695 2023-04-01 14:56:08,456 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 5ccdc27eb0704b908d36e610535e8885, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680360967695 2023-04-01 14:56:08,456 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting d87eb911a1644f2abadcd9f58247578a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1680360967721 2023-04-01 14:56:08,456 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d7c8e4d1229d4695a23bacf095693038, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1680360967721 2023-04-01 14:56:08,457 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 45db72dc0c814017b5756292cf0bed9d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680360967835 2023-04-01 14:56:08,457 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting e9bd859fff7b4ac99ec01c86d28c48bf, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680360967835 2023-04-01 14:56:08,463 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#B#compaction#564 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:08,467 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#A#compaction#563 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:08,510 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/d21d9f71e8a94901841f9416295defeb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d21d9f71e8a94901841f9416295defeb 2023-04-01 14:56:08,514 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/A of d5537299121eaceb1e94c26142bdabf4 into d21d9f71e8a94901841f9416295defeb(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:08,514 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:08,514 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/A, priority=13, startTime=1680360968453; duration=0sec 2023-04-01 14:56:08,514 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:08,514 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-01 14:56:08,515 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-01 14:56:08,515 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/C is initiating minor compaction (all files) 2023-04-01 14:56:08,515 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/C in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,515 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44bcad13ffe24e46bd7cc6e5f71aaa0b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/db8586053c2f4a9199f70d9b9e4dda7d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/938ddb9e82c64e9ea36f78bd90d4d7ca] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=34.8 K 2023-04-01 14:56:08,516 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 44bcad13ffe24e46bd7cc6e5f71aaa0b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680360967695 2023-04-01 14:56:08,516 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting db8586053c2f4a9199f70d9b9e4dda7d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1680360967721 2023-04-01 14:56:08,516 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 938ddb9e82c64e9ea36f78bd90d4d7ca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680360967835 2023-04-01 14:56:08,521 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#C#compaction#565 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:08,522 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:08,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:08,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:08,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:08,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:08,526 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:08,526 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:08,526 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:08,526 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:08,527 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,527 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,528 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,528 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:08,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:08,529 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,529 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:08,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:08,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:08,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:08,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:08,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:08,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:08,531 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/f835066826c542d9a318df89fd567b6c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f835066826c542d9a318df89fd567b6c 2023-04-01 14:56:08,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:08,531 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:08,531 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:08,531 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:08,531 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 14:56:08,531 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:08,531 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:08,531 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:08,531 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:08,531 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:08,532 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:08,535 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/C of d5537299121eaceb1e94c26142bdabf4 into f835066826c542d9a318df89fd567b6c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:08,535 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:08,535 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/C, priority=13, startTime=1680360968453; duration=0sec 2023-04-01 14:56:08,535 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:08,547 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040155cf2adbd350469e98f273b7bdedd343 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040155cf2adbd350469e98f273b7bdedd343 2023-04-01 14:56:08,551 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=71, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/f0573b5473d04bdfb5de3dfe355f6c50 2023-04-01 14:56:08,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:08,574 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:08,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361028585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361028585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361028587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361028588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361028588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361028689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361028689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361028690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361028692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361028692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,886 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/7ff83ee95a60400db3515daf128fc8c0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/7ff83ee95a60400db3515daf128fc8c0 2023-04-01 14:56:08,890 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/B of d5537299121eaceb1e94c26142bdabf4 into 7ff83ee95a60400db3515daf128fc8c0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:08,891 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:08,891 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/B, priority=13, startTime=1680360968453; duration=0sec 2023-04-01 14:56:08,891 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:08,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361028891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361028893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361028893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361028894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:08,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361028895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:08,962 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/e5b31082b4244f039135cd376ec9ee0f 2023-04-01 14:56:09,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361029198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361029199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361029200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361029201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361029201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,372 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/22a76f3b5ffe482086f230ef6e09ed7d 2023-04-01 14:56:09,376 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/f0573b5473d04bdfb5de3dfe355f6c50 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/f0573b5473d04bdfb5de3dfe355f6c50 2023-04-01 14:56:09,379 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/f0573b5473d04bdfb5de3dfe355f6c50, entries=150, sequenceid=71, filesize=25.1 K 2023-04-01 14:56:09,380 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/e5b31082b4244f039135cd376ec9ee0f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e5b31082b4244f039135cd376ec9ee0f 2023-04-01 14:56:09,383 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e5b31082b4244f039135cd376ec9ee0f, entries=150, sequenceid=71, filesize=11.6 K 2023-04-01 14:56:09,384 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/22a76f3b5ffe482086f230ef6e09ed7d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/22a76f3b5ffe482086f230ef6e09ed7d 2023-04-01 14:56:09,386 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/22a76f3b5ffe482086f230ef6e09ed7d, entries=150, sequenceid=71, filesize=11.6 K 2023-04-01 14:56:09,387 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for d5537299121eaceb1e94c26142bdabf4 in 856ms, sequenceid=71, compaction requested=false 2023-04-01 14:56:09,387 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:09,387 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:09,387 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:09,387 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:09,387 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,387 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:09,388 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:09,391 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:09,391 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:09,391 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:09,391 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:09,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:09,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:09,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c2a24d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:09,393 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:09,393 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:09,394 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:09,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:09,394 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,395 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:09,396 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:09,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:09,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:09,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:09,398 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:09,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:09,399 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:09,399 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:09,399 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:09,400 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:09,400 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:09,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:09,400 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:09,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:09,401 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,403 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:09,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:09,406 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:09,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:09,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:09,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:09,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:09,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:09,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:09,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:09,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:09,407 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:09,407 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:09,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:09,507 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:09,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:09,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:09,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:09,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:09,510 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:09,510 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:09,510 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:09,510 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:09,510 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:09,510 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:09,511 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,511 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:09,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:09,512 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,512 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:09,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:09,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:09,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:09,513 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:09,513 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:09,513 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:09,513 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:09,513 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:09,513 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:09,513 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:09,513 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 14:56:09,513 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:09,514 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:09,514 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:09,514 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:09,514 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:09,514 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:09,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:09,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:09,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361029711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361029713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361029718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361029718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361029721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361029819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361029822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361029822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361029822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:09,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361029823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:09,929 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304018bf3fd2783ba446784c28e8fb0ed4d3b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018bf3fd2783ba446784c28e8fb0ed4d3b 2023-04-01 14:56:09,933 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=89, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/03b514f2737a4b4b94131ea0102085da 2023-04-01 14:56:09,950 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/e63839d4f7c34628811f81430dec3ccd 2023-04-01 14:56:10,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361030023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361030024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361030024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361030026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361030026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,241 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-01 14:56:10,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361030327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361030327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361030327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361030330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361030330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,359 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/170c3dbb93e541c89380d062ddb9c1de 2023-04-01 14:56:10,364 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/03b514f2737a4b4b94131ea0102085da as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/03b514f2737a4b4b94131ea0102085da 2023-04-01 14:56:10,367 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/03b514f2737a4b4b94131ea0102085da, entries=150, sequenceid=89, filesize=25.1 K 2023-04-01 14:56:10,368 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/e63839d4f7c34628811f81430dec3ccd as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e63839d4f7c34628811f81430dec3ccd 2023-04-01 14:56:10,371 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e63839d4f7c34628811f81430dec3ccd, entries=150, sequenceid=89, filesize=11.6 K 2023-04-01 14:56:10,372 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/170c3dbb93e541c89380d062ddb9c1de as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/170c3dbb93e541c89380d062ddb9c1de 2023-04-01 14:56:10,376 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/170c3dbb93e541c89380d062ddb9c1de, entries=150, sequenceid=89, filesize=11.6 K 2023-04-01 14:56:10,377 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for d5537299121eaceb1e94c26142bdabf4 in 864ms, sequenceid=89, compaction requested=true 2023-04-01 14:56:10,377 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:10,377 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:10,377 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:10,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:10,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:10,379 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:10,379 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:10,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:10,379 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:10,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:10,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:10,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:10,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:10,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3387d1d3[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:10,381 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:10,381 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:10,382 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:10,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:10,382 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:10,382 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:10,382 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:10,382 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,382 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:10,383 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:10,383 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:10,383 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:10,383 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:10,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:10,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:10,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:10,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:10,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:10,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:10,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:10,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:10,386 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:10,386 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:10,386 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:10,386 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:10,386 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:10,388 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:10,388 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:10,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:10,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:10,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:10,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:10,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:10,388 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:10,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:10,390 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,391 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:10,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,393 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,393 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:10,393 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:10,393 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,393 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:10,393 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:10,393 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:10,393 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:10,393 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:10,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:10,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:10,393 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:10,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:10,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:10,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:10,394 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:10,394 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:10,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:10,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:10,494 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:10,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:10,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:10,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:10,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:10,497 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:10,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:10,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:10,497 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:10,498 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:10,498 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:10,499 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:10,499 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:10,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:10,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:10,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:10,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:10,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:10,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:10,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:10,500 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:10,500 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:10,500 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:10,501 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:10,501 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:10,501 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:10,501 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:10,501 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 14:56:10,501 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:10,501 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:10,501 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:10,501 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:10,501 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:10,501 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:10,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:10,833 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:10,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361030840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361030841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361030841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361030842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361030842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361030946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361030946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361030946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:10,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361030946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361030946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:10,949 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040134a2b0caab93425a9558adfc7df893f0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040134a2b0caab93425a9558adfc7df893f0 2023-04-01 14:56:11,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361031149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361031149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361031150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361031150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361031150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,354 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=106, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/48aefbd1311b44e5ac98d956e27d7453 2023-04-01 14:56:11,366 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/53ff2aed5cf04a5f8aed209f727e9d96 2023-04-01 14:56:11,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361031452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361031453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361031453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361031454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361031455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,776 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/b13faa0600de460b84b2066a917fe7bd 2023-04-01 14:56:11,779 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/48aefbd1311b44e5ac98d956e27d7453 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/48aefbd1311b44e5ac98d956e27d7453 2023-04-01 14:56:11,783 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/48aefbd1311b44e5ac98d956e27d7453, entries=150, sequenceid=106, filesize=25.1 K 2023-04-01 14:56:11,784 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/53ff2aed5cf04a5f8aed209f727e9d96 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/53ff2aed5cf04a5f8aed209f727e9d96 2023-04-01 14:56:11,791 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/53ff2aed5cf04a5f8aed209f727e9d96, entries=150, sequenceid=106, filesize=11.6 K 2023-04-01 14:56:11,791 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/b13faa0600de460b84b2066a917fe7bd as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/b13faa0600de460b84b2066a917fe7bd 2023-04-01 14:56:11,795 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/b13faa0600de460b84b2066a917fe7bd, entries=150, sequenceid=106, filesize=11.6 K 2023-04-01 14:56:11,795 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d5537299121eaceb1e94c26142bdabf4 in 1294ms, sequenceid=106, compaction requested=true 2023-04-01 14:56:11,795 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:11,795 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:11,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,797 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:11,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:11,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:11,798 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:11,798 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a3b136e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:11,799 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:11,799 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:11,802 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:11,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:11,802 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:11,802 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:11,802 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:11,803 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,803 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:11,803 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,803 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:11,804 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:11,804 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:11,806 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:11,806 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:11,806 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:11,807 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:11,807 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:11,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:11,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:11,811 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:11,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:11,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:11,812 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:11,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:11,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,814 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,817 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,817 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:11,817 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:11,817 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:11,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:11,817 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:11,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:11,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:11,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:11,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:11,817 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,817 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:11,817 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:11,818 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:11,818 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:11,818 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:11,818 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:11,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:11,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:11,918 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:11,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:11,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:11,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:11,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:11,921 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:11,922 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:11,922 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:11,922 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:11,922 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:11,922 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:11,923 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,923 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:11,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:11,923 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,923 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:11,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:11,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:11,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:11,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:11,925 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:11,925 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:11,925 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:11,925 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:11,925 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:11,925 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:11,925 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:11,926 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-01 14:56:11,926 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:11,926 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:11,926 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:11,926 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:11,926 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:11,926 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:11,938 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401ae16207424cf4af4a654a679d751f677 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ae16207424cf4af4a654a679d751f677 2023-04-01 14:56:11,943 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=121, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/c9d6012403b34eafae22a867a7584630 2023-04-01 14:56:11,951 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/3ffd866b88fc4eed8e55058ff2742310 2023-04-01 14:56:11,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:11,959 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:11,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361031971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361031973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361031973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361031973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:11,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:11,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361031974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361032076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361032079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361032079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361032079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361032079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361032278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361032280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361032281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361032281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361032282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,362 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/f5a76459ce264494b2d73340e3c5f8a6 2023-04-01 14:56:12,367 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/c9d6012403b34eafae22a867a7584630 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/c9d6012403b34eafae22a867a7584630 2023-04-01 14:56:12,370 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/c9d6012403b34eafae22a867a7584630, entries=150, sequenceid=121, filesize=25.1 K 2023-04-01 14:56:12,371 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/3ffd866b88fc4eed8e55058ff2742310 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3ffd866b88fc4eed8e55058ff2742310 2023-04-01 14:56:12,375 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3ffd866b88fc4eed8e55058ff2742310, entries=150, sequenceid=121, filesize=11.6 K 2023-04-01 14:56:12,376 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/f5a76459ce264494b2d73340e3c5f8a6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f5a76459ce264494b2d73340e3c5f8a6 2023-04-01 14:56:12,379 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f5a76459ce264494b2d73340e3c5f8a6, entries=150, sequenceid=121, filesize=11.6 K 2023-04-01 14:56:12,380 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d5537299121eaceb1e94c26142bdabf4 in 455ms, sequenceid=121, compaction requested=true 2023-04-01 14:56:12,380 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:12,380 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:12,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:12,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:12,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:12,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:12,381 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:12,382 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:12,382 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:12,382 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:12,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:12,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:12,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d544db4[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:12,383 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:12,383 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:12,384 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:12,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:12,385 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:12,385 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:12,385 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,385 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:12,387 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:12,387 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,387 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:12,387 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:12,389 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:12,389 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:12,389 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:12,389 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:12,389 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:12,391 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:12,391 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:12,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:12,391 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:12,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:12,392 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:12,393 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:12,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:12,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:12,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:12,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:12,395 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:12,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:12,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:12,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:12,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:12,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:12,396 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:12,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:12,396 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:12,396 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:12,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:12,496 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:12,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:12,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:12,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:12,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:12,501 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:12,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:12,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:12,502 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:12,502 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:12,502 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:12,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:12,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:12,503 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:12,504 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,504 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:12,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:12,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:12,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:12,505 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:12,505 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:12,505 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 14:56:12,505 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:12,506 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:12,506 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:12,506 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:12,506 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:12,506 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:12,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:12,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:12,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361032617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361032618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361032618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361032620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361032621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361032721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361032722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361032722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361032724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361032725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,925 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304013dc675cb0e3c47bcbb2dfc88ae354143 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013dc675cb0e3c47bcbb2dfc88ae354143 2023-04-01 14:56:12,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361032923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361032925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361032926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361032929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:12,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:12,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361032930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:13,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361033228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:13,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361033229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:13,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361033230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:13,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361033233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:13,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361033233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,330 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=138, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/5f6a85c4651246769a6c9207bf565714 2023-04-01 14:56:13,339 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/75f7d379aa9c4e559c17b811aacaefa6 2023-04-01 14:56:13,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:13,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361033733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:13,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361033734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:13,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361033734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:13,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361033739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:13,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361033741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,749 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/ad792a1a106c4ed6ba3992e3183f76ca 2023-04-01 14:56:13,752 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/5f6a85c4651246769a6c9207bf565714 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5f6a85c4651246769a6c9207bf565714 2023-04-01 14:56:13,755 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5f6a85c4651246769a6c9207bf565714, entries=150, sequenceid=138, filesize=25.3 K 2023-04-01 14:56:13,756 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/75f7d379aa9c4e559c17b811aacaefa6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/75f7d379aa9c4e559c17b811aacaefa6 2023-04-01 14:56:13,760 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/75f7d379aa9c4e559c17b811aacaefa6, entries=150, sequenceid=138, filesize=11.8 K 2023-04-01 14:56:13,761 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/ad792a1a106c4ed6ba3992e3183f76ca as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/ad792a1a106c4ed6ba3992e3183f76ca 2023-04-01 14:56:13,764 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/ad792a1a106c4ed6ba3992e3183f76ca, entries=150, sequenceid=138, filesize=11.8 K 2023-04-01 14:56:13,765 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for d5537299121eaceb1e94c26142bdabf4 in 1260ms, sequenceid=138, compaction requested=true 2023-04-01 14:56:13,765 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:13,765 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:13,765 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:13,765 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:13,765 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,765 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:13,765 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:13,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:13,768 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:13,768 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:13,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:13,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:13,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:13,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:13,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:13,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c055299[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:13,770 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:13,770 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:13,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:13,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:13,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:13,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:13,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:13,771 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:13,771 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,771 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:13,772 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,772 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:13,773 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:13,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:13,773 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:13,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:13,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:13,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:13,776 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:13,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:13,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:13,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:13,776 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:13,777 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:13,777 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:13,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:13,777 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:13,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:13,778 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:13,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:13,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:13,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:13,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:13,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:13,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:13,783 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:13,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:13,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:13,782 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:13,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:13,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:13,783 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:13,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:13,783 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:13,883 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:13,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:13,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:13,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:13,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:13,888 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:13,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:13,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:13,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:13,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:13,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:13,890 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:13,890 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:13,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:13,890 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:13,890 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:13,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:13,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:13,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:13,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:13,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:13,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:13,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:13,891 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:13,892 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:13,892 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:13,892 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:13,892 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:13,892 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:13,892 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:13,892 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-01 14:56:13,892 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:13,893 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:13,893 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:13,893 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:13,893 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:13,893 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:13,913 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401ddafc908e959479b92adc260009778ef to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ddafc908e959479b92adc260009778ef 2023-04-01 14:56:14,320 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=154, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/19385fa9f688465a9cc48493e2b5c0b7 2023-04-01 14:56:14,734 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/d1869b4e375347108dd3fab26ee14627 2023-04-01 14:56:14,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:14,740 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:14,746 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/2872b4bd9e20456790c14bf7d85f0fad 2023-04-01 14:56:14,749 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/19385fa9f688465a9cc48493e2b5c0b7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/19385fa9f688465a9cc48493e2b5c0b7 2023-04-01 14:56:14,753 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/19385fa9f688465a9cc48493e2b5c0b7, entries=150, sequenceid=154, filesize=25.3 K 2023-04-01 14:56:14,753 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/d1869b4e375347108dd3fab26ee14627 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d1869b4e375347108dd3fab26ee14627 2023-04-01 14:56:14,757 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d1869b4e375347108dd3fab26ee14627, entries=150, sequenceid=154, filesize=11.8 K 2023-04-01 14:56:14,758 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/2872b4bd9e20456790c14bf7d85f0fad as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2872b4bd9e20456790c14bf7d85f0fad 2023-04-01 14:56:14,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:14,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361034752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,761 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2872b4bd9e20456790c14bf7d85f0fad, entries=150, sequenceid=154, filesize=11.8 K 2023-04-01 14:56:14,761 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for d5537299121eaceb1e94c26142bdabf4 in 869ms, sequenceid=154, compaction requested=true 2023-04-01 14:56:14,761 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:14,761 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:14,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,763 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:14,763 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-01 14:56:14,764 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,764 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:14,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:14,765 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,765 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:14,765 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:14,765 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:14,765 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:14,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:14,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21810783[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:14,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:14,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,767 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:14,767 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,767 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:14,767 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:14,767 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:14,774 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,774 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:14,774 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:14,774 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:14,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:14,778 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:14,778 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:14,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:14,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:14,779 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:14,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:14,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:14,781 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401d27604c0b9b642c3b84ed083095f70e3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401d27604c0b9b642c3b84ed083095f70e3 2023-04-01 14:56:14,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:14,781 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361034776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:14,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361034777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361034778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,784 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:14,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361034781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:14,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,789 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:14,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:14,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:14,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:14,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:14,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:14,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:14,789 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:14,789 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:14,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:14,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:14,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361034860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:14,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361034882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:14,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361034883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:14,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361034883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:14,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361034887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,889 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:14,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:14,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:14,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:14,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:14,895 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:14,895 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:14,895 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:14,895 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:14,896 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,896 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:14,898 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:14,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:14,898 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:14,898 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,898 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:14,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:14,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:14,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:14,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:14,899 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:14,899 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:14,899 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:14,899 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:14,899 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:14,899 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:14,899 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:14,900 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:14,900 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:14,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:14,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:14,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:14,900 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:14,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,901 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:14,901 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,901 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:14,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1313c0be[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:14,903 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:14,903 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,904 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,904 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:14,904 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:14,904 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:14,904 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,904 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:14,905 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:14,906 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:14,906 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:14,906 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:14,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:14,908 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:14,908 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:14,908 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:14,908 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:14,908 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:14,911 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,911 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:14,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,911 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:14,912 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:14,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,914 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:14,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:14,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:14,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:14,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:14,916 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:14,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:14,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:14,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:14,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:14,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:14,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:14,917 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:14,917 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:14,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:15,017 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:15,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:15,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:15,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:15,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:15,020 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:15,020 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:15,020 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:15,020 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:15,021 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,021 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,022 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:15,022 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:15,022 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,022 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:15,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:15,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:15,023 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:15,023 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:15,023 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:15,024 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:15,024 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:15,024 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:15,024 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:15,024 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:15,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:15,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,026 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:15,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:15,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ca51685[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:15,028 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:15,028 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,030 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,030 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:15,030 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:15,030 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:15,030 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,030 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:15,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,032 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:15,032 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:15,032 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:15,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:15,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:15,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:15,034 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:15,034 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:15,034 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:15,034 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:15,034 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:15,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:15,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:15,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,037 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,039 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:15,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:15,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:15,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:15,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:15,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:15,040 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:15,040 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:15,039 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:15,040 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,040 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:15,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361035063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361035086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361035086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361035087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361035093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,140 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:15,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:15,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:15,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:15,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:15,143 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:15,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:15,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:15,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:15,144 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,144 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,146 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:15,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:15,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:15,146 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,146 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:15,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:15,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:15,148 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:15,148 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:15,148 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:15,148 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,149 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:15,149 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:15,150 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,150 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:15,150 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,150 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab20e12[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:15,152 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:15,152 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,154 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:15,154 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:15,154 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:15,154 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,154 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:15,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,155 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:15,155 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:15,155 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:15,158 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:15,158 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:15,158 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:15,158 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:15,158 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:15,160 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,160 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,160 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:15,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,161 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:15,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,162 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,167 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:15,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:15,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:15,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:15,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:15,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:15,167 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:15,167 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:15,167 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,168 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:15,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:15,189 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=172, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/e67e8805beb8478ea198391d847540f6 2023-04-01 14:56:15,200 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/1281e45408f14f35ab812edd28eab5c0 2023-04-01 14:56:15,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/89a680c2209a46d7adf3ffac36ac6653 2023-04-01 14:56:15,213 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/e67e8805beb8478ea198391d847540f6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e67e8805beb8478ea198391d847540f6 2023-04-01 14:56:15,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e67e8805beb8478ea198391d847540f6, entries=250, sequenceid=172, filesize=38.9 K 2023-04-01 14:56:15,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/1281e45408f14f35ab812edd28eab5c0 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1281e45408f14f35ab812edd28eab5c0 2023-04-01 14:56:15,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1281e45408f14f35ab812edd28eab5c0, entries=150, sequenceid=172, filesize=11.8 K 2023-04-01 14:56:15,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/89a680c2209a46d7adf3ffac36ac6653 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/89a680c2209a46d7adf3ffac36ac6653 2023-04-01 14:56:15,224 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/89a680c2209a46d7adf3ffac36ac6653, entries=150, sequenceid=172, filesize=11.8 K 2023-04-01 14:56:15,225 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d5537299121eaceb1e94c26142bdabf4 in 462ms, sequenceid=172, compaction requested=true 2023-04-01 14:56:15,225 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:15,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:15,225 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-01 14:56:15,225 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-01 14:56:15,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:15,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:15,227 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 220505 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-01 14:56:15,227 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95641 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-01 14:56:15,227 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/A is initiating minor compaction (all files) 2023-04-01 14:56:15,227 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/B is initiating minor compaction (all files) 2023-04-01 14:56:15,227 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/A in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:15,227 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/B in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:15,227 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d21d9f71e8a94901841f9416295defeb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/f0573b5473d04bdfb5de3dfe355f6c50, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/03b514f2737a4b4b94131ea0102085da, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/48aefbd1311b44e5ac98d956e27d7453, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/c9d6012403b34eafae22a867a7584630, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5f6a85c4651246769a6c9207bf565714, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/19385fa9f688465a9cc48493e2b5c0b7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e67e8805beb8478ea198391d847540f6] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=215.3 K 2023-04-01 14:56:15,227 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/7ff83ee95a60400db3515daf128fc8c0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e5b31082b4244f039135cd376ec9ee0f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e63839d4f7c34628811f81430dec3ccd, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/53ff2aed5cf04a5f8aed209f727e9d96, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3ffd866b88fc4eed8e55058ff2742310, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/75f7d379aa9c4e559c17b811aacaefa6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d1869b4e375347108dd3fab26ee14627, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1281e45408f14f35ab812edd28eab5c0] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=93.4 K 2023-04-01 14:56:15,228 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d21d9f71e8a94901841f9416295defeb, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680360967835 2023-04-01 14:56:15,228 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 7ff83ee95a60400db3515daf128fc8c0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680360967835 2023-04-01 14:56:15,228 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting f0573b5473d04bdfb5de3dfe355f6c50, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1680360967963 2023-04-01 14:56:15,228 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e5b31082b4244f039135cd376ec9ee0f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1680360967963 2023-04-01 14:56:15,228 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 03b514f2737a4b4b94131ea0102085da, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680360968582 2023-04-01 14:56:15,229 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e63839d4f7c34628811f81430dec3ccd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680360968582 2023-04-01 14:56:15,229 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 48aefbd1311b44e5ac98d956e27d7453, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680360969707 2023-04-01 14:56:15,229 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 53ff2aed5cf04a5f8aed209f727e9d96, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680360969707 2023-04-01 14:56:15,229 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting c9d6012403b34eafae22a867a7584630, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680360970839 2023-04-01 14:56:15,230 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3ffd866b88fc4eed8e55058ff2742310, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680360970839 2023-04-01 14:56:15,230 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 5f6a85c4651246769a6c9207bf565714, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680360971971 2023-04-01 14:56:15,230 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 75f7d379aa9c4e559c17b811aacaefa6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680360971971 2023-04-01 14:56:15,230 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 19385fa9f688465a9cc48493e2b5c0b7, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680360972615 2023-04-01 14:56:15,230 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting d1869b4e375347108dd3fab26ee14627, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680360972615 2023-04-01 14:56:15,231 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting e67e8805beb8478ea198391d847540f6, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680360974750 2023-04-01 14:56:15,231 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1281e45408f14f35ab812edd28eab5c0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680360974757 2023-04-01 14:56:15,246 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#B#compaction#588 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:15,248 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#A#compaction#587 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:15,268 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:15,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:15,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:15,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:15,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:15,277 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:15,277 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:15,277 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:15,277 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:15,277 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,277 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,279 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:15,279 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:15,279 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,279 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:15,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:15,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:15,292 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:15,292 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:15,292 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:15,292 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:15,292 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:56:15,292 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:15,293 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:15,293 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:15,293 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:15,293 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:15,293 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:15,312 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304013618684a8fc443ea88571048c1e08416 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013618684a8fc443ea88571048c1e08416 2023-04-01 14:56:15,317 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=186, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/a5f33c5ef15746d381f5d99cb7bdfd76 2023-04-01 14:56:15,325 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/5f8a06e578b541929e8b4d83024d1927 2023-04-01 14:56:15,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:15,374 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:15,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361035401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361035405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361035405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361035406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361035407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361035508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361035515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361035515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361035515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361035515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,656 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/6a825289214440f2b0f5157989f4226d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/6a825289214440f2b0f5157989f4226d 2023-04-01 14:56:15,660 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/B of d5537299121eaceb1e94c26142bdabf4 into 6a825289214440f2b0f5157989f4226d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:15,660 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:15,660 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/B, priority=8, startTime=1680360975225; duration=0sec 2023-04-01 14:56:15,661 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:15,661 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-01 14:56:15,662 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95641 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-01 14:56:15,663 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/C is initiating minor compaction (all files) 2023-04-01 14:56:15,663 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/C in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:15,663 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f835066826c542d9a318df89fd567b6c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/22a76f3b5ffe482086f230ef6e09ed7d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/170c3dbb93e541c89380d062ddb9c1de, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/b13faa0600de460b84b2066a917fe7bd, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f5a76459ce264494b2d73340e3c5f8a6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/ad792a1a106c4ed6ba3992e3183f76ca, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2872b4bd9e20456790c14bf7d85f0fad, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/89a680c2209a46d7adf3ffac36ac6653] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=93.4 K 2023-04-01 14:56:15,663 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f835066826c542d9a318df89fd567b6c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680360967835 2023-04-01 14:56:15,663 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 22a76f3b5ffe482086f230ef6e09ed7d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1680360967963 2023-04-01 14:56:15,664 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 170c3dbb93e541c89380d062ddb9c1de, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680360968582 2023-04-01 14:56:15,664 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting b13faa0600de460b84b2066a917fe7bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680360969707 2023-04-01 14:56:15,664 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f5a76459ce264494b2d73340e3c5f8a6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1680360970839 2023-04-01 14:56:15,664 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ad792a1a106c4ed6ba3992e3183f76ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680360971971 2023-04-01 14:56:15,665 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2872b4bd9e20456790c14bf7d85f0fad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680360972615 2023-04-01 14:56:15,665 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 89a680c2209a46d7adf3ffac36ac6653, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680360974757 2023-04-01 14:56:15,676 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#C#compaction#592 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:15,688 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/de9345c40c8f43b2b060b9b01f9a1c7b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/de9345c40c8f43b2b060b9b01f9a1c7b 2023-04-01 14:56:15,694 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/C of d5537299121eaceb1e94c26142bdabf4 into de9345c40c8f43b2b060b9b01f9a1c7b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:15,694 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:15,694 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/C, priority=8, startTime=1680360975225; duration=0sec 2023-04-01 14:56:15,694 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:15,700 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/cc33c609b12d456bb956de4c6d3c7579 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/cc33c609b12d456bb956de4c6d3c7579 2023-04-01 14:56:15,705 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/A of d5537299121eaceb1e94c26142bdabf4 into cc33c609b12d456bb956de4c6d3c7579(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:15,705 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:15,705 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/A, priority=8, startTime=1680360975225; duration=0sec 2023-04-01 14:56:15,705 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:15,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361035712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361035719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361035719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361035719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:15,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361035720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,738 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/44870f955c9f47dd91dfb27f186cbae2 2023-04-01 14:56:15,742 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/a5f33c5ef15746d381f5d99cb7bdfd76 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/a5f33c5ef15746d381f5d99cb7bdfd76 2023-04-01 14:56:15,745 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/a5f33c5ef15746d381f5d99cb7bdfd76, entries=150, sequenceid=186, filesize=25.3 K 2023-04-01 14:56:15,746 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/5f8a06e578b541929e8b4d83024d1927 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/5f8a06e578b541929e8b4d83024d1927 2023-04-01 14:56:15,749 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/5f8a06e578b541929e8b4d83024d1927, entries=150, sequenceid=186, filesize=11.8 K 2023-04-01 14:56:15,750 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/44870f955c9f47dd91dfb27f186cbae2 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44870f955c9f47dd91dfb27f186cbae2 2023-04-01 14:56:15,753 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44870f955c9f47dd91dfb27f186cbae2, entries=150, sequenceid=186, filesize=11.8 K 2023-04-01 14:56:15,754 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d5537299121eaceb1e94c26142bdabf4 in 462ms, sequenceid=186, compaction requested=false 2023-04-01 14:56:15,754 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:15,754 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:15,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:15,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:15,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:15,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:15,755 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,755 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:15,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:15,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:15,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:15,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:15,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:15,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3326afd8[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:15,757 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:15,757 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,759 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:15,759 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:15,759 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:15,759 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,759 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:15,760 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,770 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:15,770 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:15,770 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:15,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:15,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:15,772 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:15,773 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:15,773 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:15,773 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:15,773 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:15,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:15,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:15,776 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:15,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,778 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:15,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:15,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:15,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:15,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:15,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,780 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:15,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:15,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:15,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:15,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:15,780 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:15,781 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:15,780 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:15,781 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:15,881 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:15,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:15,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:15,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:15,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:15,885 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:15,885 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:15,885 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:15,885 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:15,885 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,886 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,887 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:15,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,887 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:15,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:15,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:15,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:15,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:15,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:15,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:15,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:15,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:15,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:15,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:15,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:15,889 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:15,889 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:15,889 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:15,890 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:56:15,890 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:15,890 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:15,890 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:15,890 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:15,890 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:15,890 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:16,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:16,015 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:16,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361036027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361036027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361036028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361036028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361036029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361036130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361036130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361036132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361036132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361036132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,307 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401fd0d66aa6c3446b1a78d27ef34c77cf4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401fd0d66aa6c3446b1a78d27ef34c77cf4 2023-04-01 14:56:16,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361036333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361036334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361036335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361036336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361036336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361036637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361036639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361036639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361036639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:16,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361036640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,712 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=207, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/40e7339460f1480a8d7848eccc5e3d3e 2023-04-01 14:56:16,726 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/cfc4ee8a684644898646eecb69899f92 2023-04-01 14:56:16,738 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/f83b144fbd854068b9e672169eb54e4c 2023-04-01 14:56:16,742 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/40e7339460f1480a8d7848eccc5e3d3e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/40e7339460f1480a8d7848eccc5e3d3e 2023-04-01 14:56:16,745 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/40e7339460f1480a8d7848eccc5e3d3e, entries=150, sequenceid=207, filesize=25.3 K 2023-04-01 14:56:16,746 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/cfc4ee8a684644898646eecb69899f92 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cfc4ee8a684644898646eecb69899f92 2023-04-01 14:56:16,749 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cfc4ee8a684644898646eecb69899f92, entries=150, sequenceid=207, filesize=11.8 K 2023-04-01 14:56:16,749 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/f83b144fbd854068b9e672169eb54e4c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f83b144fbd854068b9e672169eb54e4c 2023-04-01 14:56:16,752 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f83b144fbd854068b9e672169eb54e4c, entries=150, sequenceid=207, filesize=11.8 K 2023-04-01 14:56:16,753 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d5537299121eaceb1e94c26142bdabf4 in 864ms, sequenceid=207, compaction requested=true 2023-04-01 14:56:16,753 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:16,753 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:16,755 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:16,756 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:16,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:16,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ebd8f32[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:16,757 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:16,757 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:16,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:16,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:16,760 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:16,760 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,760 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:16,760 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:16,760 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:16,761 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:16,762 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:16,762 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:16,762 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:16,765 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:16,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:16,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:16,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:16,765 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:16,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:16,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:16,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:16,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:16,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:16,767 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:16,769 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:16,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:16,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:16,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:16,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:16,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:16,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:16,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,771 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:16,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:16,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:16,771 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:16,771 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:16,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:16,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:16,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:16,871 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:16,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:16,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:16,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:16,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:16,875 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:16,875 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:16,875 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:16,875 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:16,876 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:16,876 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:16,878 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:16,878 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:16,878 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:16,878 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:16,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:16,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:16,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:16,879 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:16,879 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:16,880 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:16,880 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:16,880 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:16,880 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:16,880 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:16,880 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:56:16,880 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:16,880 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:16,880 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:16,880 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:16,881 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:16,881 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:17,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:17,142 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:17,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361037154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361037154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361037157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361037158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361037158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361037259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361037259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361037262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361037262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361037263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,298 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401788a434aa1b146828197810c79750f3b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401788a434aa1b146828197810c79750f3b 2023-04-01 14:56:17,303 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=221, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/3c86eb3446954587a43b93259222bcc7 2023-04-01 14:56:17,312 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/eb93f1dd1a1040eba392d0e6c8d3021c 2023-04-01 14:56:17,322 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/10e3e3dafdd8452e9e23c721e66d4d31 2023-04-01 14:56:17,327 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/3c86eb3446954587a43b93259222bcc7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/3c86eb3446954587a43b93259222bcc7 2023-04-01 14:56:17,330 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/3c86eb3446954587a43b93259222bcc7, entries=150, sequenceid=221, filesize=25.3 K 2023-04-01 14:56:17,331 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/eb93f1dd1a1040eba392d0e6c8d3021c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/eb93f1dd1a1040eba392d0e6c8d3021c 2023-04-01 14:56:17,334 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/eb93f1dd1a1040eba392d0e6c8d3021c, entries=150, sequenceid=221, filesize=11.8 K 2023-04-01 14:56:17,335 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/10e3e3dafdd8452e9e23c721e66d4d31 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/10e3e3dafdd8452e9e23c721e66d4d31 2023-04-01 14:56:17,349 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/10e3e3dafdd8452e9e23c721e66d4d31, entries=150, sequenceid=221, filesize=11.8 K 2023-04-01 14:56:17,350 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for d5537299121eaceb1e94c26142bdabf4 in 470ms, sequenceid=221, compaction requested=true 2023-04-01 14:56:17,350 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:17,350 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,350 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:17,350 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:17,350 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,350 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:17,350 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:17,351 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,351 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,352 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,352 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:17,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:17,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a18b75e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:17,354 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:17,354 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,355 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,355 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:17,355 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:17,355 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:17,356 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,356 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:17,357 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:17,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:17,360 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:17,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:17,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:17,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:17,360 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:17,362 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,362 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,362 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,363 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,364 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,367 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:17,367 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:17,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:17,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:17,367 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:17,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:17,367 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:17,367 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:17,367 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:17,367 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,367 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:17,367 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,367 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,367 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,368 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:17,463 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 14:56:17,464 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:17,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:17,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:17,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:17,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:17,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:17,467 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:17,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:17,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:17,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:17,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:17,471 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:17,471 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:17,471 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:17,471 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:17,471 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,472 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,473 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:17,473 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:17,473 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,473 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:17,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:17,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:17,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:17,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:17,475 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,475 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:17,475 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:17,475 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:17,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:17,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:17,475 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:17,477 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401d82e581e12a54b5d82d4c567b532bd86 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401d82e581e12a54b5d82d4c567b532bd86 2023-04-01 14:56:17,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361037473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361037477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361037479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361037479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361037480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,485 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,485 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,485 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,485 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:17,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50ea91[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:17,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:17,487 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,488 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,488 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:17,488 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:17,488 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,488 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:17,488 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:17,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,490 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:17,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:17,492 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:17,492 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:17,492 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:17,492 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:17,492 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:17,493 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,493 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:17,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,493 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:17,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,498 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:17,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,503 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:17,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,503 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:17,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:17,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:17,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:17,503 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:17,503 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:17,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:17,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361037580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361037583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361037585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361037585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361037585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,603 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:17,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:17,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:17,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:17,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:17,607 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:17,607 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:17,607 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:17,607 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:17,607 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,607 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,609 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,609 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:17,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:17,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:17,609 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,609 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:17,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:17,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:17,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:17,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:17,611 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,611 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:17,611 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:17,611 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:17,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,612 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,612 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,612 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:17,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60ecd2c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:17,614 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:17,614 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,615 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:17,616 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,616 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:17,621 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,621 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:17,621 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:17,621 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:17,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:17,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:17,624 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:17,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:17,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:17,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:17,624 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:17,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:17,626 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,627 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,629 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,629 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:17,629 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:17,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:17,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:17,629 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,629 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:17,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:17,629 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:17,630 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,630 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:17,630 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:17,630 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:17,630 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,630 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,730 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:17,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:17,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:17,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:17,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:17,740 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:17,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:17,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:17,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:17,741 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,741 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,742 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,742 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:17,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:17,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:17,742 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,742 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:17,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:17,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:17,744 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:17,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:17,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:17,745 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,745 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:17,745 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:17,745 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:17,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:17,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:17,745 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:17,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,753 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,754 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:17,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:17,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@549806c0[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:17,755 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:17,755 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,756 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,757 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:17,757 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:17,757 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:17,757 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,757 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:17,758 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:17,758 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:17,758 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:17,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:17,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:17,761 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:17,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:17,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:17,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:17,761 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:17,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:17,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:17,764 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,765 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:17,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:17,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,769 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:17,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:17,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:17,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:17,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,769 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:17,770 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:17,770 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:17,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:17,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361037785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41784 deadline: 1680361037785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361037789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361037789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:17,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361037790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,870 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:17,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:17,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:17,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:17,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:17,873 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:17,873 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:17,873 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:17,873 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:17,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,874 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,875 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:17,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:17,875 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:17,875 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,875 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:17,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:17,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:17,877 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,877 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:17,877 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:17,877 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:17,877 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:17,878 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,878 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,879 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,879 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:17,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf5f94a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:17,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:17,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,882 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=242, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/e914c686bf924e4c8509a8c7a2e8df31 2023-04-01 14:56:17,886 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,886 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,886 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:17,886 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:17,886 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:17,886 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:17,888 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,888 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:17,888 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:17,888 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/cc4d6faa774941eba193c508949d055b 2023-04-01 14:56:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:17,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:17,892 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:17,892 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:17,892 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:17,892 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:17,892 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:17,895 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,895 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:17,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,895 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:17,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:17,896 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:17,898 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,900 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:17,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:17,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:17,901 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:17,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:17,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:17,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:17,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:17,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:17,901 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:17,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:17,902 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/bb84e8fb92f34b27851b22b8d3958535 2023-04-01 14:56:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:17,902 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:17,902 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:17,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:17,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/e914c686bf924e4c8509a8c7a2e8df31 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e914c686bf924e4c8509a8c7a2e8df31 2023-04-01 14:56:17,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e914c686bf924e4c8509a8c7a2e8df31, entries=200, sequenceid=242, filesize=32.1 K 2023-04-01 14:56:17,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/cc4d6faa774941eba193c508949d055b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cc4d6faa774941eba193c508949d055b 2023-04-01 14:56:17,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cc4d6faa774941eba193c508949d055b, entries=150, sequenceid=242, filesize=11.8 K 2023-04-01 14:56:17,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/bb84e8fb92f34b27851b22b8d3958535 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/bb84e8fb92f34b27851b22b8d3958535 2023-04-01 14:56:17,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/bb84e8fb92f34b27851b22b8d3958535, entries=150, sequenceid=242, filesize=11.8 K 2023-04-01 14:56:17,919 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for d5537299121eaceb1e94c26142bdabf4 in 456ms, sequenceid=242, compaction requested=true 2023-04-01 14:56:17,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:17,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:17,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:17,919 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:56:17,919 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:56:17,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:17,921 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60556 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:56:17,921 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/B is initiating minor compaction (all files) 2023-04-01 14:56:17,921 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136846 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:56:17,921 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/B in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,921 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/A is initiating minor compaction (all files) 2023-04-01 14:56:17,921 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/A in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,921 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/6a825289214440f2b0f5157989f4226d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/5f8a06e578b541929e8b4d83024d1927, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cfc4ee8a684644898646eecb69899f92, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/eb93f1dd1a1040eba392d0e6c8d3021c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cc4d6faa774941eba193c508949d055b] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=59.1 K 2023-04-01 14:56:17,921 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/cc33c609b12d456bb956de4c6d3c7579, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/a5f33c5ef15746d381f5d99cb7bdfd76, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/40e7339460f1480a8d7848eccc5e3d3e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/3c86eb3446954587a43b93259222bcc7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e914c686bf924e4c8509a8c7a2e8df31] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=133.6 K 2023-04-01 14:56:17,922 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6a825289214440f2b0f5157989f4226d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680360974757 2023-04-01 14:56:17,922 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting cc33c609b12d456bb956de4c6d3c7579, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680360974757 2023-04-01 14:56:17,922 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 5f8a06e578b541929e8b4d83024d1927, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680360974770 2023-04-01 14:56:17,922 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting a5f33c5ef15746d381f5d99cb7bdfd76, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680360974770 2023-04-01 14:56:17,922 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cfc4ee8a684644898646eecb69899f92, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680360975398 2023-04-01 14:56:17,922 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 40e7339460f1480a8d7848eccc5e3d3e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680360975398 2023-04-01 14:56:17,922 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting eb93f1dd1a1040eba392d0e6c8d3021c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1680360976026 2023-04-01 14:56:17,923 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cc4d6faa774941eba193c508949d055b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680360977157 2023-04-01 14:56:17,923 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 3c86eb3446954587a43b93259222bcc7, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1680360976026 2023-04-01 14:56:17,923 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting e914c686bf924e4c8509a8c7a2e8df31, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680360977156 2023-04-01 14:56:17,932 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#B#compaction#602 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:17,935 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#A#compaction#603 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:17,948 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/85c7f4aae3bc4d71bdb8c570ab19cb2b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/85c7f4aae3bc4d71bdb8c570ab19cb2b 2023-04-01 14:56:17,952 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/B of d5537299121eaceb1e94c26142bdabf4 into 85c7f4aae3bc4d71bdb8c570ab19cb2b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:17,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:17,952 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/B, priority=11, startTime=1680360977919; duration=0sec 2023-04-01 14:56:17,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:17,952 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:56:17,954 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60556 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:56:17,954 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/C is initiating minor compaction (all files) 2023-04-01 14:56:17,954 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/C in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:17,954 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/de9345c40c8f43b2b060b9b01f9a1c7b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44870f955c9f47dd91dfb27f186cbae2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f83b144fbd854068b9e672169eb54e4c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/10e3e3dafdd8452e9e23c721e66d4d31, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/bb84e8fb92f34b27851b22b8d3958535] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=59.1 K 2023-04-01 14:56:17,955 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting de9345c40c8f43b2b060b9b01f9a1c7b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680360974757 2023-04-01 14:56:17,955 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 44870f955c9f47dd91dfb27f186cbae2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680360974770 2023-04-01 14:56:17,955 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f83b144fbd854068b9e672169eb54e4c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680360975398 2023-04-01 14:56:17,956 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 10e3e3dafdd8452e9e23c721e66d4d31, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1680360976026 2023-04-01 14:56:17,957 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting bb84e8fb92f34b27851b22b8d3958535, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680360977157 2023-04-01 14:56:17,964 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#C#compaction#604 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:18,002 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:18,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:18,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:18,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:18,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:18,005 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:18,005 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:18,005 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:18,005 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:18,006 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:18,006 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:18,008 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:18,008 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:18,008 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,008 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:18,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:18,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:18,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:18,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:18,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:18,009 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:18,010 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:18,010 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:18,010 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:18,010 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:18,010 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:18,010 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 14:56:18,010 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:18,010 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:18,010 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:18,010 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:18,010 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:18,010 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:18,025 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040120f619f4bb39458fa48ec7e2d6e11c61 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040120f619f4bb39458fa48ec7e2d6e11c61 2023-04-01 14:56:18,031 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=255, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/d190107168fb42f49dbb17b8bc6adfc9 2023-04-01 14:56:18,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:18,091 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:18,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361038105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361038106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361038106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361038107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41788 deadline: 1680361038108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680361038209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361038210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361038210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361038211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361038211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,357 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/915d3fa7b4cf422195db4a7a2c0f5549 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/915d3fa7b4cf422195db4a7a2c0f5549 2023-04-01 14:56:18,362 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/A of d5537299121eaceb1e94c26142bdabf4 into 915d3fa7b4cf422195db4a7a2c0f5549(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:18,362 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:18,362 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/A, priority=11, startTime=1680360977919; duration=0sec 2023-04-01 14:56:18,362 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:18,373 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/cbb6f39d1368496fbd05f085ae63d336 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/cbb6f39d1368496fbd05f085ae63d336 2023-04-01 14:56:18,378 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/C of d5537299121eaceb1e94c26142bdabf4 into cbb6f39d1368496fbd05f085ae63d336(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:18,379 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:18,379 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/C, priority=11, startTime=1680360977919; duration=0sec 2023-04-01 14:56:18,379 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:18,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361038413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41794 deadline: 1680361038414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361038415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361038415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41776 deadline: 1680361038415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,441 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/cf7739d148014321931607a3899babcc 2023-04-01 14:56:18,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361038717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361038718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361038718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361038719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:18,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361038719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,851 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/0e4f8d834d6b4989adf50d69dc1b7b1d 2023-04-01 14:56:18,855 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/d190107168fb42f49dbb17b8bc6adfc9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d190107168fb42f49dbb17b8bc6adfc9 2023-04-01 14:56:18,858 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d190107168fb42f49dbb17b8bc6adfc9, entries=150, sequenceid=255, filesize=25.3 K 2023-04-01 14:56:18,859 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/cf7739d148014321931607a3899babcc as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cf7739d148014321931607a3899babcc 2023-04-01 14:56:18,862 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cf7739d148014321931607a3899babcc, entries=150, sequenceid=255, filesize=11.8 K 2023-04-01 14:56:18,863 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/0e4f8d834d6b4989adf50d69dc1b7b1d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/0e4f8d834d6b4989adf50d69dc1b7b1d 2023-04-01 14:56:18,866 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/0e4f8d834d6b4989adf50d69dc1b7b1d, entries=150, sequenceid=255, filesize=11.8 K 2023-04-01 14:56:18,867 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d5537299121eaceb1e94c26142bdabf4 in 857ms, sequenceid=255, compaction requested=false 2023-04-01 14:56:18,867 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:18,867 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:18,869 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:18,869 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:18,869 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:18,869 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:18,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:18,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cf69cc7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:18,871 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:18,871 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:18,872 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:18,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:18,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:18,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:18,872 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:18,872 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,872 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:18,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:18,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:18,874 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:18,874 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:18,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:18,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:18,877 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:18,877 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:18,877 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:18,877 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:18,877 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:18,880 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:18,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:18,880 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:18,880 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:18,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:18,881 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:18,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,882 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:18,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:18,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,885 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,885 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:18,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:18,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:18,885 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:18,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:18,885 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:18,885 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:18,885 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:18,885 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:18,885 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:18,885 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:18,885 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:18,885 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:18,885 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:18,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:18,986 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:18,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:18,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:18,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:18,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:18,989 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:18,989 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:18,989 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:18,989 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:18,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:18,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:18,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,991 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:18,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:18,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:18,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:18,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:18,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:18,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:18,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:18,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:18,993 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:18,993 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:18,993 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-01 14:56:18,994 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:18,994 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:18,994 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:18,994 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:18,994 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:18,994 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:19,007 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401df233e0004cb43379dde95e163596db6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401df233e0004cb43379dde95e163596db6 2023-04-01 14:56:19,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:19,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:19,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361039229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361039230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361039231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361039231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361039233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361039334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361039335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361039335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361039335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361039337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,413 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=277, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/88fb8e21311a40a3a1e760cff5901908 2023-04-01 14:56:19,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361039537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361039537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361039537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361039538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361039538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,823 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/798382e17d014b71a79f0c56b8e8d4fa 2023-04-01 14:56:19,834 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/e3d3f38c512e4da089d3f935a98dc349 2023-04-01 14:56:19,838 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/88fb8e21311a40a3a1e760cff5901908 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/88fb8e21311a40a3a1e760cff5901908 2023-04-01 14:56:19,841 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/88fb8e21311a40a3a1e760cff5901908, entries=150, sequenceid=277, filesize=25.4 K 2023-04-01 14:56:19,842 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/798382e17d014b71a79f0c56b8e8d4fa as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/798382e17d014b71a79f0c56b8e8d4fa 2023-04-01 14:56:19,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361039841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361039842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361039842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361039842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:19,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361039843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,846 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/798382e17d014b71a79f0c56b8e8d4fa, entries=150, sequenceid=277, filesize=11.9 K 2023-04-01 14:56:19,847 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/e3d3f38c512e4da089d3f935a98dc349 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/e3d3f38c512e4da089d3f935a98dc349 2023-04-01 14:56:19,850 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/e3d3f38c512e4da089d3f935a98dc349, entries=150, sequenceid=277, filesize=11.9 K 2023-04-01 14:56:19,850 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d5537299121eaceb1e94c26142bdabf4 in 857ms, sequenceid=277, compaction requested=true 2023-04-01 14:56:19,850 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:19,850 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:19,850 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:19,851 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:19,851 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,851 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:19,851 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:19,852 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:19,852 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:19,852 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:19,853 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:19,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:19,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c6155d8[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:19,854 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:19,854 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:19,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:19,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:19,856 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:19,856 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:19,856 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:19,857 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,857 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:19,858 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,858 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:19,858 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:19,858 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:19,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:19,861 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:19,861 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:19,861 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:19,861 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:19,861 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:19,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:19,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:19,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:19,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:19,863 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,865 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:19,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:19,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:19,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:19,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:19,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:19,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:19,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:19,868 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:19,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:19,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:19,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:19,868 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:19,868 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:19,969 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:19,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:19,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:19,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:19,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:19,972 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:19,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:19,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:19,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:19,973 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:19,973 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:19,975 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:19,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:19,975 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:19,975 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:19,976 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:19,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:19,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:19,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:19,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:19,977 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:19,977 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:19,977 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:19,977 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:19,978 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:19,978 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:19,978 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 14:56:19,978 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:19,978 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:19,978 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:19,978 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:19,978 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:19,978 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:20,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:20,346 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361040360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361040361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361040361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361040362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361040364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,393 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304015b6414e89e1a4fdf8fa4ff9bfe3c2607 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015b6414e89e1a4fdf8fa4ff9bfe3c2607 2023-04-01 14:56:20,405 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=289, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/5b43a0a32d594a3ab0eac01fa48849da 2023-04-01 14:56:20,416 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/8aeae02b878f43ccb7172a3f69fe9166 2023-04-01 14:56:20,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361040465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361040466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361040466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361040466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361040468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361040669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361040670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361040670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361040670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:20,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361040678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,834 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/df160d1ad4d648eaad59e3d623e4092a 2023-04-01 14:56:20,837 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/5b43a0a32d594a3ab0eac01fa48849da as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5b43a0a32d594a3ab0eac01fa48849da 2023-04-01 14:56:20,840 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5b43a0a32d594a3ab0eac01fa48849da, entries=150, sequenceid=289, filesize=25.4 K 2023-04-01 14:56:20,840 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/8aeae02b878f43ccb7172a3f69fe9166 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/8aeae02b878f43ccb7172a3f69fe9166 2023-04-01 14:56:20,844 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/8aeae02b878f43ccb7172a3f69fe9166, entries=150, sequenceid=289, filesize=11.9 K 2023-04-01 14:56:20,844 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/df160d1ad4d648eaad59e3d623e4092a as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/df160d1ad4d648eaad59e3d623e4092a 2023-04-01 14:56:20,847 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/df160d1ad4d648eaad59e3d623e4092a, entries=150, sequenceid=289, filesize=11.9 K 2023-04-01 14:56:20,848 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d5537299121eaceb1e94c26142bdabf4 in 870ms, sequenceid=289, compaction requested=true 2023-04-01 14:56:20,848 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:20,848 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:20,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:20,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:20,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:20,848 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:20,850 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78dbe092[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:20,852 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:20,852 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,853 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:20,853 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,853 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:20,855 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,855 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:20,855 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:20,855 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:20,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:20,857 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:20,857 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:20,857 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:20,857 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:20,858 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:20,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,863 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:20,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:20,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:20,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:20,864 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,865 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:20,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:20,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:20,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:20,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:20,867 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:20,867 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:20,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,868 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:20,868 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:20,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,868 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:20,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:20,968 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:20,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:20,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:20,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:20,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:20,972 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:20,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:20,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:20,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:20,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:20,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:20,973 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:20,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:20,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360980358, max timestamp=1680360980971Num uniques 859; 2023-04-01 14:56:20,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:56:20,974 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-04-01 14:56:20,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360980358, max timestamp=1680360980971Num uniques 859; 2023-04-01 14:56:20,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:56:20,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:20,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:20,974 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360980358, max timestamp=1680360980971Num uniques 859; 2023-04-01 14:56:20,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:56:20,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:20,975 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:20,975 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,975 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:20,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,975 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:20,975 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:20,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:20,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:20,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:20,975 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:20,976 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:20,976 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:20,976 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:20,976 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:20,978 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:20,978 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:20,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:20,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:20,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c50696a[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:20,981 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:20,981 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,983 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:20,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:20,983 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:20,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:20,985 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:20,985 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:20,985 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:20,985 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:20,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:20,988 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:20,988 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:20,988 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:20,988 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:20,988 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:20,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:20,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:20,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,990 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:20,990 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:20,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:20,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,992 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,994 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:20,994 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:20,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:20,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:20,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:20,994 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:20,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:20,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:20,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:20,995 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:20,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:20,995 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:20,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:20,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:20,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:20,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:20,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680360980994, max timestamp=1680360980996Num uniques 859; 2023-04-01 14:56:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680360980994, max timestamp=1680360980996Num uniques 859; 2023-04-01 14:56:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680360980994, max timestamp=1680360980996Num uniques 859; 2023-04-01 14:56:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:20,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:21,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680360981009, max timestamp=1680360981015Num uniques 1068; 2023-04-01 14:56:21,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-01 14:56:21,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680360981009, max timestamp=1680360981015Num uniques 1068; 2023-04-01 14:56:21,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-01 14:56:21,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680360981009, max timestamp=1680360981015Num uniques 1068; 2023-04-01 14:56:21,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-01 14:56:21,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680360981027, max timestamp=1680360981033Num uniques 1127; 2023-04-01 14:56:21,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-01 14:56:21,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680360981027, max timestamp=1680360981033Num uniques 1127; 2023-04-01 14:56:21,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-01 14:56:21,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680360981027, max timestamp=1680360981033Num uniques 1127; 2023-04-01 14:56:21,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-01 14:56:21,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680360981045, max timestamp=1680360981051Num uniques 1186; 2023-04-01 14:56:21,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-01 14:56:21,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680360981045, max timestamp=1680360981051Num uniques 1186; 2023-04-01 14:56:21,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-01 14:56:21,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680360981045, max timestamp=1680360981051Num uniques 1186; 2023-04-01 14:56:21,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-01 14:56:21,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680360981062, max timestamp=1680360981070Num uniques 1095; 2023-04-01 14:56:21,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-01 14:56:21,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680360981062, max timestamp=1680360981070Num uniques 1095; 2023-04-01 14:56:21,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-01 14:56:21,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680360981062, max timestamp=1680360981070Num uniques 1095; 2023-04-01 14:56:21,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-01 14:56:21,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13914, totalHeapSize=31856, min timestamp=1680360981084, max timestamp=1680360981092Num uniques 1104; 2023-04-01 14:56:21,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=13914, suffix heap size=142784,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=304 2023-04-01 14:56:21,095 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:21,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13914, totalHeapSize=31856, min timestamp=1680360981084, max timestamp=1680360981092Num uniques 1104; 2023-04-01 14:56:21,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=13914, suffix heap size=142784,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=304 2023-04-01 14:56:21,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13914, totalHeapSize=31856, min timestamp=1680360981084, max timestamp=1680360981092Num uniques 1104; 2023-04-01 14:56:21,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=13914, suffix heap size=142784,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=304 2023-04-01 14:56:21,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:21,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:21,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:21,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:21,100 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:21,100 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,100 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:21,100 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:21,100 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,100 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,102 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,103 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:21,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:21,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:21,104 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,105 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:21,105 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:21,105 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:21,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:21,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:21,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:21,116 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:21,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e8d483[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:21,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:21,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,122 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:21,123 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,123 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:21,124 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:21,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:21,127 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:21,127 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:21,127 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:21,127 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:21,127 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:21,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680360981117, max timestamp=1680360981128Num uniques 1163; 2023-04-01 14:56:21,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680360981117, max timestamp=1680360981128Num uniques 1163; 2023-04-01 14:56:21,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680360981117, max timestamp=1680360981128Num uniques 1163; 2023-04-01 14:56:21,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=9748, suffix heap size=148920,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=213 2023-04-01 14:56:21,130 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=9748, suffix heap size=148920,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=213 2023-04-01 14:56:21,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,131 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,131 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=9748, suffix heap size=148920,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=213 2023-04-01 14:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,132 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,133 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,136 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:21,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:21,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,136 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:21,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:21,136 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:21,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680360981142, max timestamp=1680360981148Num uniques 1072; 2023-04-01 14:56:21,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-01 14:56:21,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680360981142, max timestamp=1680360981148Num uniques 1072; 2023-04-01 14:56:21,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-01 14:56:21,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680360981142, max timestamp=1680360981148Num uniques 1072; 2023-04-01 14:56:21,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-01 14:56:21,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680360981159, max timestamp=1680360981164Num uniques 1081; 2023-04-01 14:56:21,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-01 14:56:21,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680360981159, max timestamp=1680360981164Num uniques 1081; 2023-04-01 14:56:21,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680360981159, max timestamp=1680360981164Num uniques 1081; 2023-04-01 14:56:21,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-01 14:56:21,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-01 14:56:21,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680360981176, max timestamp=1680360981183Num uniques 1090; 2023-04-01 14:56:21,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=17860, suffix heap size=141328,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=390 2023-04-01 14:56:21,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680360981176, max timestamp=1680360981183Num uniques 1090; 2023-04-01 14:56:21,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=17860, suffix heap size=141328,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=390 2023-04-01 14:56:21,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680360981176, max timestamp=1680360981183Num uniques 1090; 2023-04-01 14:56:21,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=17860, suffix heap size=141328,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=390 2023-04-01 14:56:21,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361041200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361041200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1680360981196, max timestamp=1680360981203Num uniques 1249; 2023-04-01 14:56:21,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1680360981196, max timestamp=1680360981203Num uniques 1249; 2023-04-01 14:56:21,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680360981197, max timestamp=1680360981203Num uniques 1249; 2023-04-01 14:56:21,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=11404, suffix heap size=157864,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=249 2023-04-01 14:56:21,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=15984, suffix heap size=157864,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=349 2023-04-01 14:56:21,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=15984, suffix heap size=157864,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=349 2023-04-01 14:56:21,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361041229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680360981222, max timestamp=1680360981232Num uniques 1208; 2023-04-01 14:56:21,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680360981222, max timestamp=1680360981232Num uniques 1208; 2023-04-01 14:56:21,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680360981222, max timestamp=1680360981232Num uniques 1108; 2023-04-01 14:56:21,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=55320, new segment data size=9520, suffix heap size=153600,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=208 2023-04-01 14:56:21,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-01 14:56:21,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55320, new segment data size=9520, suffix heap size=153600,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=208 2023-04-01 14:56:21,236 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:21,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:21,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:21,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:21,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:21,240 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:21,240 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,240 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:21,240 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:21,241 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,241 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,242 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,242 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,242 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,242 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:21,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:21,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:21,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:21,244 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:21,244 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,244 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:21,244 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:21,244 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:21,247 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,247 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,247 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,247 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:21,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c19253[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:21,249 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:21,249 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,251 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:21,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,252 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:21,253 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:21,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,254 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,254 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:21,257 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:21,257 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:21,257 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:21,257 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:21,257 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:21,259 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,259 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,259 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,260 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,261 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,268 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:21,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:21,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:21,268 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:21,268 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:21,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680360981257, max timestamp=1680360981273Num uniques 1067; 2023-04-01 14:56:21,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-01 14:56:21,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680360981257, max timestamp=1680360981273Num uniques 1067; 2023-04-01 14:56:21,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-01 14:56:21,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680360981257, max timestamp=1680360981273Num uniques 1067; 2023-04-01 14:56:21,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-01 14:56:21,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12633, totalHeapSize=28944, min timestamp=1680360981296, max timestamp=1680360981311Num uniques 1026; 2023-04-01 14:56:21,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12631, totalHeapSize=28944, min timestamp=1680360981296, max timestamp=1680360981311Num uniques 1026; 2023-04-01 14:56:21,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46981, new segment data size=12631, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-04-01 14:56:21,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12631, totalHeapSize=28944, min timestamp=1680360981296, max timestamp=1680360981311Num uniques 1026; 2023-04-01 14:56:21,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46983, new segment data size=12633, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-04-01 14:56:21,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46981, new segment data size=12631, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-04-01 14:56:21,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680360981324, max timestamp=1680360981331Num uniques 1135; 2023-04-01 14:56:21,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=13050, suffix heap size=146008,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=285 2023-04-01 14:56:21,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680360981324, max timestamp=1680360981331Num uniques 1135; 2023-04-01 14:56:21,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=13050, suffix heap size=146008,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=285 2023-04-01 14:56:21,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1680360981324, max timestamp=1680360981331Num uniques 1135; 2023-04-01 14:56:21,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=8470, suffix heap size=146008,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=185 2023-04-01 14:56:21,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680360981352, max timestamp=1680360981358Num uniques 1144; 2023-04-01 14:56:21,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680360981352, max timestamp=1680360981358Num uniques 1144; 2023-04-01 14:56:21,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680360981352, max timestamp=1680360981358Num uniques 1044; 2023-04-01 14:56:21,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-04-01 14:56:21,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-01 14:56:21,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-04-01 14:56:21,369 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:21,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:21,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:21,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:21,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:21,373 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:21,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:21,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:21,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,374 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,375 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,375 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,375 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:21,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:21,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:21,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680360981377, max timestamp=1680360981384Num uniques 1053; 2023-04-01 14:56:21,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-01 14:56:21,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1680360981377, max timestamp=1680360981384Num uniques 1053; 2023-04-01 14:56:21,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-01 14:56:21,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1680360981377, max timestamp=1680360981384Num uniques 1053; 2023-04-01 14:56:21,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-01 14:56:21,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:21,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:21,396 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,396 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:21,396 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:21,396 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:21,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:21,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:21,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:21,398 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,398 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,399 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,400 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401afe12bbac3184868b74b60f3840640dc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401afe12bbac3184868b74b60f3840640dc 2023-04-01 14:56:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:21,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25393de9[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:21,401 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:21,402 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,403 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,403 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:21,403 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:21,403 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:21,403 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,403 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:21,406 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,408 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=310, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/ae917c1cb389464a90af32d762073e02 2023-04-01 14:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:21,409 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:21,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:21,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:21,409 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:21,409 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:21,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,411 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1680360981401, max timestamp=1680360981410Num uniques 1062; 2023-04-01 14:56:21,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-01 14:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1680360981401, max timestamp=1680360981410Num uniques 1012; 2023-04-01 14:56:21,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-01 14:56:21,413 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1680360981401, max timestamp=1680360981410Num uniques 1012; 2023-04-01 14:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-01 14:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,414 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,416 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,418 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:21,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:21,418 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:21,419 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:21,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/3aae5ceffbaf4c248202bdde9670470c 2023-04-01 14:56:21,426 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,426 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680360981422, max timestamp=1680360981436Num uniques 1121; 2023-04-01 14:56:21,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-04-01 14:56:21,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680360981422, max timestamp=1680360981436Num uniques 1121; 2023-04-01 14:56:21,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680360981422, max timestamp=1680360981436Num uniques 1121; 2023-04-01 14:56:21,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-04-01 14:56:21,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-04-01 14:56:21,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680360981444, max timestamp=1680360981455Num uniques 1030; 2023-04-01 14:56:21,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680360981444, max timestamp=1680360981455Num uniques 1030; 2023-04-01 14:56:21,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=12820, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-01 14:56:21,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680360981450, max timestamp=1680360981455Num uniques 1030; 2023-04-01 14:56:21,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=12820, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-01 14:56:21,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-01 14:56:21,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1680360981463, max timestamp=1680360981497Num uniques 1139; 2023-04-01 14:56:21,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=8654, suffix heap size=146424,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=189 2023-04-01 14:56:21,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1680360981463, max timestamp=1680360981497Num uniques 1139; 2023-04-01 14:56:21,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=8654, suffix heap size=146424,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=189 2023-04-01 14:56:21,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1680360981463, max timestamp=1680360981497Num uniques 1089; 2023-04-01 14:56:21,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-01 14:56:21,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680360981504, max timestamp=1680360981515Num uniques 1048; 2023-04-01 14:56:21,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-01 14:56:21,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680360981504, max timestamp=1680360981515Num uniques 1048; 2023-04-01 14:56:21,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-01 14:56:21,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680360981504, max timestamp=1680360981515Num uniques 1048; 2023-04-01 14:56:21,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-01 14:56:21,519 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:21,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:21,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:21,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:21,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:21,522 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:21,522 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,522 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:21,522 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:21,522 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,523 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,524 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,524 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,524 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,524 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:21,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:21,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:21,526 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,526 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:21,526 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:21,526 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:21,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:21,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:21,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:21,528 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aed1b1d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:21,531 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:21,531 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,532 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,532 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:21,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:21,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:21,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:21,534 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,534 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:21,534 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,534 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:21,538 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:21,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:21,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:21,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:21,538 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:21,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=307, cellSize=14055, totalHeapSize=32168, min timestamp=1680360981528, max timestamp=1680360981536Num uniques 1107; 2023-04-01 14:56:21,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=14055, suffix heap size=143096,new segment heap size=32168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=307 2023-04-01 14:56:21,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=307, cellSize=14055, totalHeapSize=32168, min timestamp=1680360981528, max timestamp=1680360981536Num uniques 1107; 2023-04-01 14:56:21,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=14055, suffix heap size=143096,new segment heap size=32168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=307 2023-04-01 14:56:21,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1680360981528, max timestamp=1680360981536Num uniques 1107; 2023-04-01 14:56:21,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=7185, suffix heap size=143096,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=157 2023-04-01 14:56:21,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,541 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,543 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:21,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:21,546 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,546 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:21,546 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:21,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,547 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=316, cellSize=14466, totalHeapSize=33104, min timestamp=1680360981547, max timestamp=1680360981554Num uniques 1166; 2023-04-01 14:56:21,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53396, new segment data size=14466, suffix heap size=149232,new segment heap size=33104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1166, new segment cells count=316 2023-04-01 14:56:21,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=316, cellSize=14466, totalHeapSize=33104, min timestamp=1680360981547, max timestamp=1680360981554Num uniques 1016; 2023-04-01 14:56:21,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=14466, suffix heap size=133632,new segment heap size=33104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=316 2023-04-01 14:56:21,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=316, cellSize=14466, totalHeapSize=33104, min timestamp=1680360981547, max timestamp=1680360981554Num uniques 1166; 2023-04-01 14:56:21,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53396, new segment data size=14466, suffix heap size=149232,new segment heap size=33104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1166, new segment cells count=316 2023-04-01 14:56:21,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361041571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361041571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361041572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361041572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1680360981564, max timestamp=1680360981574Num uniques 1175; 2023-04-01 14:56:21,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53810, new segment data size=12590, suffix heap size=150168,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1175, new segment cells count=275 2023-04-01 14:56:21,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1680360981564, max timestamp=1680360981574Num uniques 1175; 2023-04-01 14:56:21,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53810, new segment data size=12590, suffix heap size=150168,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1175, new segment cells count=275 2023-04-01 14:56:21,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1680360981564, max timestamp=1680360981574Num uniques 1175; 2023-04-01 14:56:21,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53810, new segment data size=12590, suffix heap size=150168,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1175, new segment cells count=275 2023-04-01 14:56:21,647 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:21,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:21,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:21,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:21,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:21,650 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:21,650 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,650 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:21,650 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:21,650 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,650 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,651 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,651 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,652 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,652 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:21,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:21,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:21,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:21,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1680360981622, max timestamp=1680360981652Num uniques 1134; 2023-04-01 14:56:21,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=8424, suffix heap size=145904,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=184 2023-04-01 14:56:21,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1680360981622, max timestamp=1680360981652Num uniques 1134; 2023-04-01 14:56:21,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1680360981622, max timestamp=1680360981652Num uniques 1134; 2023-04-01 14:56:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:21,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=8424, suffix heap size=145904,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=184 2023-04-01 14:56:21,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=8424, suffix heap size=145904,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=184 2023-04-01 14:56:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:21,655 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,655 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:21,656 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:21,656 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,657 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:21,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4803860f[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:21,659 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:21,659 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,661 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,661 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:21,661 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:21,661 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:21,661 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,661 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,663 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,664 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:21,666 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:21,666 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:21,666 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:21,666 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:21,666 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:21,668 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,668 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,668 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,668 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,670 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,674 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,674 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,674 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,674 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,674 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,674 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,674 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,674 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:21,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:21,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:21,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,674 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,674 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,675 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:21,675 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:21,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1680360981682, max timestamp=1680360981693Num uniques 1043; 2023-04-01 14:56:21,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=11128, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-04-01 14:56:21,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1680360981682, max timestamp=1680360981693Num uniques 1043; 2023-04-01 14:56:21,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1680360981682, max timestamp=1680360981693Num uniques 1043; 2023-04-01 14:56:21,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=11128, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-04-01 14:56:21,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=11128, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-04-01 14:56:21,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=302, cellSize=13830, totalHeapSize=31648, min timestamp=1680360981704, max timestamp=1680360981711Num uniques 1102; 2023-04-01 14:56:21,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=13830, suffix heap size=142576,new segment heap size=31648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=302 2023-04-01 14:56:21,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=302, cellSize=13830, totalHeapSize=31648, min timestamp=1680360981704, max timestamp=1680360981711Num uniques 1102; 2023-04-01 14:56:21,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=302, cellSize=13830, totalHeapSize=31648, min timestamp=1680360981704, max timestamp=1680360981711Num uniques 1102; 2023-04-01 14:56:21,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=13830, suffix heap size=142576,new segment heap size=31648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=302 2023-04-01 14:56:21,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=13830, suffix heap size=142576,new segment heap size=31648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=302 2023-04-01 14:56:21,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361041729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361041730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361041731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361041732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680360981728, max timestamp=1680360981731Num uniques 1161; 2023-04-01 14:56:21,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53166, new segment data size=9656, suffix heap size=148712,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1161, new segment cells count=211 2023-04-01 14:56:21,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680360981728, max timestamp=1680360981731Num uniques 1161; 2023-04-01 14:56:21,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53166, new segment data size=9656, suffix heap size=148712,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1161, new segment cells count=211 2023-04-01 14:56:21,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680360981728, max timestamp=1680360981731Num uniques 1161; 2023-04-01 14:56:21,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53166, new segment data size=9656, suffix heap size=148712,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1161, new segment cells count=211 2023-04-01 14:56:21,775 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:21,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:21,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:21,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:21,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:21,778 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:21,778 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,778 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:21,778 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:21,778 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,779 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,781 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,781 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,781 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,781 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:21,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:21,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:21,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:21,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:21,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:21,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:21,783 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,783 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:21,783 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:21,783 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:21,783 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:21,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:21,784 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:21,785 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,785 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,785 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,785 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:21,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@465ed656[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:21,786 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:21,787 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,787 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:21,788 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,788 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:21,789 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,789 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:21,789 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,789 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:21,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:21,793 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:21,793 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:21,793 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:21,793 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:21,793 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:21,795 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,795 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,795 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,796 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,797 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:21,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:21,799 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,799 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:21,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:21,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,800 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:21,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,800 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1680360981783, max timestamp=1680360981803Num uniques 1070; 2023-04-01 14:56:21,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1680360981783, max timestamp=1680360981803Num uniques 1070; 2023-04-01 14:56:21,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=7780, suffix heap size=139248,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=170 2023-04-01 14:56:21,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=7780, suffix heap size=139248,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=170 2023-04-01 14:56:21,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1680360981783, max timestamp=1680360981803Num uniques 1070; 2023-04-01 14:56:21,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=7780, suffix heap size=139248,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=170 2023-04-01 14:56:21,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/106702735ed24477b4eced4153640f4f 2023-04-01 14:56:21,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/ae917c1cb389464a90af32d762073e02 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/ae917c1cb389464a90af32d762073e02 2023-04-01 14:56:21,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/ae917c1cb389464a90af32d762073e02, entries=150, sequenceid=310, filesize=25.4 K 2023-04-01 14:56:21,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/3aae5ceffbaf4c248202bdde9670470c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3aae5ceffbaf4c248202bdde9670470c 2023-04-01 14:56:21,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3aae5ceffbaf4c248202bdde9670470c, entries=150, sequenceid=310, filesize=11.9 K 2023-04-01 14:56:21,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/106702735ed24477b4eced4153640f4f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/106702735ed24477b4eced4153640f4f 2023-04-01 14:56:21,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1680360981838, max timestamp=1680360981845Num uniques 1029; 2023-04-01 14:56:21,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=8194, suffix heap size=134984,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=179 2023-04-01 14:56:21,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=329, cellSize=15064, totalHeapSize=34456, min timestamp=1680360981837, max timestamp=1680360981845Num uniques 1029; 2023-04-01 14:56:21,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1680360981838, max timestamp=1680360981845Num uniques 1029; 2023-04-01 14:56:21,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=15064, suffix heap size=134984,new segment heap size=34456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=329 2023-04-01 14:56:21,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=8194, suffix heap size=134984,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=179 2023-04-01 14:56:21,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/106702735ed24477b4eced4153640f4f, entries=150, sequenceid=310, filesize=11.9 K 2023-04-01 14:56:21,851 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=33.54 KB/34350 for d5537299121eaceb1e94c26142bdabf4 in 877ms, sequenceid=310, compaction requested=true 2023-04-01 14:56:21,851 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:21,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:21,852 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:21,852 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:56:21,852 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:56:21,852 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:21,853 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130476 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:56:21,853 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61176 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:56:21,853 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/A is initiating minor compaction (all files) 2023-04-01 14:56:21,853 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/B is initiating minor compaction (all files) 2023-04-01 14:56:21,853 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/A in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,853 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/B in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,853 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/915d3fa7b4cf422195db4a7a2c0f5549, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d190107168fb42f49dbb17b8bc6adfc9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/88fb8e21311a40a3a1e760cff5901908, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5b43a0a32d594a3ab0eac01fa48849da, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/ae917c1cb389464a90af32d762073e02] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=127.4 K 2023-04-01 14:56:21,853 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/85c7f4aae3bc4d71bdb8c570ab19cb2b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cf7739d148014321931607a3899babcc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/798382e17d014b71a79f0c56b8e8d4fa, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/8aeae02b878f43ccb7172a3f69fe9166, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3aae5ceffbaf4c248202bdde9670470c] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=59.7 K 2023-04-01 14:56:21,854 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 915d3fa7b4cf422195db4a7a2c0f5549, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680360977157 2023-04-01 14:56:21,854 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 85c7f4aae3bc4d71bdb8c570ab19cb2b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680360977157 2023-04-01 14:56:21,854 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting d190107168fb42f49dbb17b8bc6adfc9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680360977475 2023-04-01 14:56:21,854 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cf7739d148014321931607a3899babcc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680360977475 2023-04-01 14:56:21,854 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 88fb8e21311a40a3a1e760cff5901908, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680360978105 2023-04-01 14:56:21,854 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 798382e17d014b71a79f0c56b8e8d4fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680360978105 2023-04-01 14:56:21,855 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 5b43a0a32d594a3ab0eac01fa48849da, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680360979228 2023-04-01 14:56:21,855 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 8aeae02b878f43ccb7172a3f69fe9166, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680360979228 2023-04-01 14:56:21,855 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting ae917c1cb389464a90af32d762073e02, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680360980358 2023-04-01 14:56:21,855 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3aae5ceffbaf4c248202bdde9670470c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680360980358 2023-04-01 14:56:21,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:21,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=53.67 KB heapSize=139.36 KB 2023-04-01 14:56:21,861 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:21,861 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:21,861 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:21,861 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:21,861 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:21,861 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:21,875 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#B#compaction#619 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:21,878 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#A#compaction#617 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:21,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361041879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680360981875, max timestamp=1680360981884Num uniques 859; 2023-04-01 14:56:21,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:21,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680360981875, max timestamp=1680360981884Num uniques 859; 2023-04-01 14:56:21,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680360981875, max timestamp=1680360981884Num uniques 859; 2023-04-01 14:56:21,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:21,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:21,895 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304012faaf7578a4741c3953b4282736b377a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012faaf7578a4741c3953b4282736b377a 2023-04-01 14:56:21,900 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:21,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:21,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:21,905 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/e522c99371c84d7288463e3d6381e2c9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e522c99371c84d7288463e3d6381e2c9 2023-04-01 14:56:21,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:21,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:21,905 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:21,905 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,905 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:21,905 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:21,906 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,906 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,907 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=851, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/7b60d84e122d47babc4781d7084900e6 2023-04-01 14:56:21,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,908 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,909 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,909 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:21,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:21,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:21,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:21,910 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/B of d5537299121eaceb1e94c26142bdabf4 into e522c99371c84d7288463e3d6381e2c9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:21,910 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:21,910 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/B, priority=11, startTime=1680360981852; duration=0sec 2023-04-01 14:56:21,911 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:21,911 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:56:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:21,912 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61176 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:56:21,912 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/C is initiating minor compaction (all files) 2023-04-01 14:56:21,912 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/C in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,912 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/cbb6f39d1368496fbd05f085ae63d336, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/0e4f8d834d6b4989adf50d69dc1b7b1d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/e3d3f38c512e4da089d3f935a98dc349, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/df160d1ad4d648eaad59e3d623e4092a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/106702735ed24477b4eced4153640f4f] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=59.7 K 2023-04-01 14:56:21,913 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cbb6f39d1368496fbd05f085ae63d336, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680360977157 2023-04-01 14:56:21,913 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 0e4f8d834d6b4989adf50d69dc1b7b1d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680360977475 2023-04-01 14:56:21,914 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e3d3f38c512e4da089d3f935a98dc349, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680360978105 2023-04-01 14:56:21,914 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting df160d1ad4d648eaad59e3d623e4092a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680360979228 2023-04-01 14:56:21,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361041907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361041908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,914 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 106702735ed24477b4eced4153640f4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680360980358 2023-04-01 14:56:21,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361041909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:21,918 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,918 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:21,918 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:21,918 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:21,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361041914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,920 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:21,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a647d01[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:21,922 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:21,922 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,924 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:21,924 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,924 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,925 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:21,928 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:21,928 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:21,928 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:21,928 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:21,928 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:21,930 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,930 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,931 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#C#compaction#621 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:21,931 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:21,931 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,932 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=851 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/58904f78b40c4dce9dbe0c6527711c8e 2023-04-01 14:56:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,933 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:21,936 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:21,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:21,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:21,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:21,937 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:21,937 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:21,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:21,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:21,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:21,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:21,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:21,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:21,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:21,947 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/ff1bb16f54b04e73b927ccd9b724ca2f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/ff1bb16f54b04e73b927ccd9b724ca2f 2023-04-01 14:56:21,951 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/C of d5537299121eaceb1e94c26142bdabf4 into ff1bb16f54b04e73b927ccd9b724ca2f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:21,951 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:21,951 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/C, priority=11, startTime=1680360981852; duration=0sec 2023-04-01 14:56:21,951 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:21,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:21,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361041986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361042015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361042015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361042015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361042020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,037 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:22,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:22,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:22,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:22,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:22,040 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:22,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:22,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:22,040 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:22,041 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,041 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,043 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:22,043 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,044 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,044 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:22,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:22,045 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:22,045 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:22,045 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:22,045 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:22,047 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,047 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:22,047 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,047 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28d3aea3[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:22,049 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:22,049 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,050 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:22,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:22,050 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:22,050 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,050 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:22,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:22,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:22,051 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:22,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:22,054 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:22,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:22,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:22,054 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:22,054 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:22,061 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,061 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:22,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,061 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:22,061 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:22,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:22,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:22,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:22,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,063 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,065 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,065 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,065 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,065 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,065 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:22,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:22,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:22,066 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:22,066 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:22,065 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:22,065 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:22,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:22,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:22,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,066 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:22,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:22,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:22,166 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:22,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:22,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:22,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:22,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:22,169 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:22,169 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:22,170 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:22,170 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:22,170 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,170 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,171 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,171 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:22,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:22,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,172 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:22,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:22,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:22,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:22,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:22,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:22,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:22,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:22,173 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:22,173 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:22,173 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:22,173 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:22,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:22,173 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:22,178 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:22,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,178 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cb46648[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:22,180 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:22,180 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,181 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,181 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:22,181 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:22,181 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,181 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:22,181 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:22,183 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:22,183 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:22,183 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:22,183 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:22,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:22,185 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:22,185 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:22,185 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:22,185 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:22,185 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:22,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:22,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:22,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:22,188 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,190 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361042190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:22,193 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:22,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:22,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:22,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:22,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:22,193 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:22,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,194 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:22,194 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:22,194 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:22,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361042218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361042219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361042220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361042225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,294 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:22,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:22,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:22,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:22,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:22,298 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:22,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:22,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:22,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:22,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,299 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:22,299 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:22,300 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,300 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:22,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:22,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:22,301 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:22,301 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:22,301 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:22,301 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:22,302 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:22,302 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:22,302 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,302 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:22,302 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:22,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,303 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:22,303 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:22,304 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ce8e3ee[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:22,305 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:22,305 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,306 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:22,307 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:22,307 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:22,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,307 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:22,310 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:22,310 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:22,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:22,310 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:22,311 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/38aaab2f28ea41369ce7ad0c58848591 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/38aaab2f28ea41369ce7ad0c58848591 2023-04-01 14:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:22,313 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:22,313 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:22,313 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:22,313 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:22,313 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:22,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:22,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:22,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,316 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/A of d5537299121eaceb1e94c26142bdabf4 into 38aaab2f28ea41369ce7ad0c58848591(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:22,316 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:22,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:22,316 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:22,316 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/A, priority=11, startTime=1680360981851; duration=0sec 2023-04-01 14:56:22,316 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:22,317 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:22,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:22,319 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,320 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,320 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,320 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:22,320 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:22,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:22,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:22,320 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:22,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:22,320 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:22,320 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,320 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:22,320 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:22,320 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:22,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:22,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:22,320 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:22,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=851 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/3872819f04a54aecb82e37a5928d4659 2023-04-01 14:56:22,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/7b60d84e122d47babc4781d7084900e6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7b60d84e122d47babc4781d7084900e6 2023-04-01 14:56:22,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7b60d84e122d47babc4781d7084900e6, entries=200, sequenceid=851, filesize=32.3 K 2023-04-01 14:56:22,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/58904f78b40c4dce9dbe0c6527711c8e as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/58904f78b40c4dce9dbe0c6527711c8e 2023-04-01 14:56:22,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/58904f78b40c4dce9dbe0c6527711c8e, entries=150, sequenceid=851, filesize=11.9 K 2023-04-01 14:56:22,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/3872819f04a54aecb82e37a5928d4659 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/3872819f04a54aecb82e37a5928d4659 2023-04-01 14:56:22,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/3872819f04a54aecb82e37a5928d4659, entries=150, sequenceid=851, filesize=11.9 K 2023-04-01 14:56:22,369 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~179.16 KB/183456, currentSize=120.76 KB/123660 for d5537299121eaceb1e94c26142bdabf4 in 513ms, sequenceid=851, compaction requested=false 2023-04-01 14:56:22,369 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:22,420 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:22,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:22,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:22,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:22,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:22,424 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:22,424 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:22,424 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:22,424 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:22,424 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,425 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,426 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:22,426 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:22,426 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,426 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:22,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:22,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:22,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:22,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:22,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:22,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:22,428 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:22,428 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:22,428 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:22,428 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-01 14:56:22,429 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:22,429 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:22,429 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:22,429 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:22,429 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:22,429 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:22,442 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401ed3290173e764308a1d7bd4460e7d6d6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ed3290173e764308a1d7bd4460e7d6d6 2023-04-01 14:56:22,447 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=888, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/122433d501ce467f88d048808135f51f 2023-04-01 14:56:22,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:22,495 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:22,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361042518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361042521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361042522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361042525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361042529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361042622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:22,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361042825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:22,857 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=888 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/f6aff69c537947ceb37901d7d9ed1c5c 2023-04-01 14:56:23,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:23,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361043026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:23,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361043027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:23,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361043029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:23,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361043032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:23,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361043129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,270 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=888 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/a634f1afffb142cf8ac5607b30fa43a3 2023-04-01 14:56:23,274 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/122433d501ce467f88d048808135f51f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/122433d501ce467f88d048808135f51f 2023-04-01 14:56:23,277 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/122433d501ce467f88d048808135f51f, entries=150, sequenceid=888, filesize=25.4 K 2023-04-01 14:56:23,278 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/f6aff69c537947ceb37901d7d9ed1c5c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/f6aff69c537947ceb37901d7d9ed1c5c 2023-04-01 14:56:23,281 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/f6aff69c537947ceb37901d7d9ed1c5c, entries=150, sequenceid=888, filesize=11.9 K 2023-04-01 14:56:23,282 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/a634f1afffb142cf8ac5607b30fa43a3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/a634f1afffb142cf8ac5607b30fa43a3 2023-04-01 14:56:23,285 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/a634f1afffb142cf8ac5607b30fa43a3, entries=150, sequenceid=888, filesize=11.9 K 2023-04-01 14:56:23,285 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for d5537299121eaceb1e94c26142bdabf4 in 857ms, sequenceid=888, compaction requested=true 2023-04-01 14:56:23,286 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:23,286 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:23,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,288 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:23,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,289 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:23,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:23,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ed0307[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:23,290 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:23,290 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,291 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,291 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:23,291 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:23,291 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,292 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:23,291 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:23,293 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:23,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:23,293 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:23,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:23,296 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:23,296 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:23,296 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:23,296 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:23,296 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:23,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:23,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:23,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:23,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:23,298 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,300 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:23,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:23,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:23,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:23,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:23,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:23,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:23,303 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,303 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:23,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,304 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:23,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:23,304 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:23,404 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:23,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:23,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:23,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:23,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:23,413 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:23,413 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:23,413 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:23,413 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:23,413 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,413 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:23,415 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:23,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:23,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:23,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:23,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:23,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:23,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:23,417 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:23,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:23,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:23,417 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-01 14:56:23,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:23,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:23,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:23,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:23,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:23,417 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:23,429 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401778298f508b14453bab5a7d76cb09b28 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401778298f508b14453bab5a7d76cb09b28 2023-04-01 14:56:23,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:23,637 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:23,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:23,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361043708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:23,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361043811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,838 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=900, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/12ba21aafd1c4b8b80ba98ca0ebd0508 2023-04-01 14:56:23,856 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=900 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/d05248fd7ba740de94bd43e47eda8728 2023-04-01 14:56:23,866 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=900 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/81df411ac7c94ccaa6a53486e02fed68 2023-04-01 14:56:23,871 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/12ba21aafd1c4b8b80ba98ca0ebd0508 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/12ba21aafd1c4b8b80ba98ca0ebd0508 2023-04-01 14:56:23,875 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/12ba21aafd1c4b8b80ba98ca0ebd0508, entries=150, sequenceid=900, filesize=25.4 K 2023-04-01 14:56:23,876 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/d05248fd7ba740de94bd43e47eda8728 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d05248fd7ba740de94bd43e47eda8728 2023-04-01 14:56:23,879 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d05248fd7ba740de94bd43e47eda8728, entries=150, sequenceid=900, filesize=11.9 K 2023-04-01 14:56:23,880 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/81df411ac7c94ccaa6a53486e02fed68 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/81df411ac7c94ccaa6a53486e02fed68 2023-04-01 14:56:23,883 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/81df411ac7c94ccaa6a53486e02fed68, entries=150, sequenceid=900, filesize=11.9 K 2023-04-01 14:56:23,884 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d5537299121eaceb1e94c26142bdabf4 in 467ms, sequenceid=900, compaction requested=true 2023-04-01 14:56:23,884 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:23,884 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:23,884 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:23,886 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fe1baf[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:23,888 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:23,889 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,895 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,895 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:23,895 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:23,895 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:23,895 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,895 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:23,897 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:23,897 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,897 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:23,897 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:23,900 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:23,900 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:23,900 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:23,900 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:23,900 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:23,902 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,902 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:23,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,902 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:23,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:23,903 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,905 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:23,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:23,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:23,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:23,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:23,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:23,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:23,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:23,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:23,908 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:23,908 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:23,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:23,909 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:23,909 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:24,009 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:24,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:24,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:24,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:24,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:24,013 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:24,013 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,013 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:24,013 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:24,013 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,013 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,015 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,015 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,016 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,016 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:24,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:24,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360983668, max timestamp=1680360984015Num uniques 859; 2023-04-01 14:56:24,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360983668, max timestamp=1680360984015Num uniques 859; 2023-04-01 14:56:24,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:24,017 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:24,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:56:24,017 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,017 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-01 14:56:24,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680360983668, max timestamp=1680360984015Num uniques 859; 2023-04-01 14:56:24,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:56:24,017 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:24,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:24,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:24,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:24,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:24,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-01 14:56:24,017 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:24,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:24,018 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:24,018 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:24,018 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,018 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:24,018 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:24,018 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,018 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:24,018 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:24,018 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,019 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:24,019 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:24,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:24,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fa7a5de[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:24,023 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:24,024 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,025 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,025 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:24,025 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:24,025 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:24,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,025 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,027 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,027 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:24,029 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:24,029 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:24,029 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:24,029 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:24,029 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:24,031 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,032 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,033 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,036 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:24,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:24,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,037 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:24,037 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:24,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:24,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,037 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,037 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,037 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360984045, max timestamp=1680360984048Num uniques 859; 2023-04-01 14:56:24,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:24,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360984045, max timestamp=1680360984048Num uniques 859; 2023-04-01 14:56:24,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:24,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680360984045, max timestamp=1680360984048Num uniques 859; 2023-04-01 14:56:24,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-01 14:56:24,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680360984057, max timestamp=1680360984066Num uniques 1068; 2023-04-01 14:56:24,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-01 14:56:24,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680360984057, max timestamp=1680360984066Num uniques 1068; 2023-04-01 14:56:24,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-01 14:56:24,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680360984057, max timestamp=1680360984066Num uniques 1068; 2023-04-01 14:56:24,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-01 14:56:24,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361044086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361044087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361044087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361044087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680360984078, max timestamp=1680360984089Num uniques 1177; 2023-04-01 14:56:24,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680360984078, max timestamp=1680360984089Num uniques 1177; 2023-04-01 14:56:24,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=14972, suffix heap size=150376,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=327 2023-04-01 14:56:24,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680360984078, max timestamp=1680360984089Num uniques 1177; 2023-04-01 14:56:24,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=14972, suffix heap size=150376,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=327 2023-04-01 14:56:24,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=14972, suffix heap size=150376,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=327 2023-04-01 14:56:24,137 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:24,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:24,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:24,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:24,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:24,140 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:24,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:24,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:24,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,141 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,141 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,142 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,142 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:24,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:24,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:24,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:24,144 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:24,145 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24dbc457[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:24,147 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:24,147 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,149 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,149 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:24,149 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:24,149 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:24,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,149 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:24,150 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,150 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:24,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,151 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:24,153 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:24,153 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:24,153 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:24,153 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:24,153 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:24,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,155 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,157 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,159 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,159 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:24,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:24,159 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,159 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,159 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:24,159 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:24,159 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,159 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,159 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,159 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,159 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,159 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:24,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680360984145, max timestamp=1680360984159Num uniques 1186; 2023-04-01 14:56:24,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-04-01 14:56:24,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680360984145, max timestamp=1680360984159Num uniques 1186; 2023-04-01 14:56:24,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680360984145, max timestamp=1680360984159Num uniques 1186; 2023-04-01 14:56:24,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-04-01 14:56:24,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-04-01 14:56:24,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11216, totalHeapSize=25720, min timestamp=1680360984196, max timestamp=1680360984201Num uniques 1045; 2023-04-01 14:56:24,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47856, new segment data size=11216, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-01 14:56:24,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680360984196, max timestamp=1680360984200Num uniques 1045; 2023-04-01 14:56:24,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11210, totalHeapSize=25720, min timestamp=1680360984196, max timestamp=1680360984201Num uniques 1045; 2023-04-01 14:56:24,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-01 14:56:24,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=11210, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-01 14:56:24,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1680360984214, max timestamp=1680360984220Num uniques 1104; 2023-04-01 14:56:24,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=9334, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-01 14:56:24,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1680360984214, max timestamp=1680360984220Num uniques 1104; 2023-04-01 14:56:24,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=9334, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-01 14:56:24,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680360984214, max timestamp=1680360984220Num uniques 1104; 2023-04-01 14:56:24,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=6870, suffix heap size=142784,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=150 2023-04-01 14:56:24,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680360984243, max timestamp=1680360984247Num uniques 1063; 2023-04-01 14:56:24,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680360984243, max timestamp=1680360984247Num uniques 1063; 2023-04-01 14:56:24,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-01 14:56:24,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-01 14:56:24,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680360984243, max timestamp=1680360984247Num uniques 1009; 2023-04-01 14:56:24,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7458, suffix heap size=132904,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=163 2023-04-01 14:56:24,259 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:24,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:24,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:24,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:24,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:24,262 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:24,262 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,262 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:24,262 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:24,263 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,263 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,265 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,265 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,265 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:24,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:24,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:24,266 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:24,266 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:24,267 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,267 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:24,267 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:24,267 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:24,267 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:24,268 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,269 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:24,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9031ef0[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:24,270 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:24,270 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680360984264, max timestamp=1680360984271Num uniques 1022; 2023-04-01 14:56:24,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-01 14:56:24,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680360984264, max timestamp=1680360984271Num uniques 1022; 2023-04-01 14:56:24,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-01 14:56:24,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680360984264, max timestamp=1680360984271Num uniques 1022; 2023-04-01 14:56:24,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-01 14:56:24,274 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,274 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:24,274 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:24,274 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:24,274 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,274 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:24,275 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,275 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:24,278 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:24,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:24,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:24,278 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:24,278 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:24,284 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,284 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,284 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,285 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,286 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,287 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,288 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:24,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:24,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:24,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,288 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:24,288 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:24,288 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680360984284, max timestamp=1680360984293Num uniques 1081; 2023-04-01 14:56:24,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-01 14:56:24,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680360984284, max timestamp=1680360984293Num uniques 1081; 2023-04-01 14:56:24,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-01 14:56:24,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680360984284, max timestamp=1680360984293Num uniques 1081; 2023-04-01 14:56:24,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=6870, suffix heap size=140392,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=150 2023-04-01 14:56:24,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680360984305, max timestamp=1680360984313Num uniques 1140; 2023-04-01 14:56:24,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-01 14:56:24,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680360984305, max timestamp=1680360984313Num uniques 1090; 2023-04-01 14:56:24,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-01 14:56:24,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680360984305, max timestamp=1680360984313Num uniques 1009; 2023-04-01 14:56:24,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10990, suffix heap size=132904,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=240 2023-04-01 14:56:24,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680360984326, max timestamp=1680360984333Num uniques 1099; 2023-04-01 14:56:24,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-01 14:56:24,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680360984326, max timestamp=1680360984333Num uniques 1099; 2023-04-01 14:56:24,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11394, totalHeapSize=26136, min timestamp=1680360984326, max timestamp=1680360984333Num uniques 1099; 2023-04-01 14:56:24,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-01 14:56:24,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50324, new segment data size=11394, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-01 14:56:24,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1680360984370, max timestamp=1680360984377Num uniques 1108; 2023-04-01 14:56:24,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=14100, suffix heap size=143200,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=308 2023-04-01 14:56:24,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1680360984370, max timestamp=1680360984377Num uniques 1108; 2023-04-01 14:56:24,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1680360984370, max timestamp=1680360984377Num uniques 1108; 2023-04-01 14:56:24,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=14100, suffix heap size=143200,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=308 2023-04-01 14:56:24,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=14100, suffix heap size=143200,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=308 2023-04-01 14:56:24,388 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:24,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:24,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:24,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:24,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:24,391 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:24,391 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,391 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:24,391 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:24,391 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,391 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,392 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,393 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,393 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,393 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:24,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:24,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:24,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:24,394 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,394 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:24,394 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:24,394 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:24,394 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,395 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:24,395 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:24,396 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,396 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:24,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9204f1c[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:24,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:24,398 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,400 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,400 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,401 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:24,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361044397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361044397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361044397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361044398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680360984387, max timestamp=1680360984399Num uniques 1167; 2023-04-01 14:56:24,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=9932, suffix heap size=149336,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=217 2023-04-01 14:56:24,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680360984387, max timestamp=1680360984399Num uniques 1167; 2023-04-01 14:56:24,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=9932, suffix heap size=149336,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=217 2023-04-01 14:56:24,403 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:24,403 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:24,403 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:24,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680360984387, max timestamp=1680360984399Num uniques 1167; 2023-04-01 14:56:24,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=9932, suffix heap size=149336,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=217 2023-04-01 14:56:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:24,404 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,404 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:24,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:24,407 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:24,407 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:24,407 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:24,407 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:24,407 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:24,409 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,409 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,409 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,413 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,415 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,417 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:24,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:24,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,418 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:24,418 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:24,431 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401b8018d3d909043e1a98bc5fcb110faf1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b8018d3d909043e1a98bc5fcb110faf1 2023-04-01 14:56:24,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680360984428, max timestamp=1680360984481Num uniques 1076; 2023-04-01 14:56:24,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680360984428, max timestamp=1680360984481Num uniques 1076; 2023-04-01 14:56:24,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-01 14:56:24,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680360984428, max timestamp=1680360984481Num uniques 1076; 2023-04-01 14:56:24,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-01 14:56:24,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-01 14:56:24,518 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:24,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:24,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:24,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:24,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:24,521 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:24,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:24,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:24,522 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,522 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680360984509, max timestamp=1680360984521Num uniques 1035; 2023-04-01 14:56:24,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680360984509, max timestamp=1680360984521Num uniques 1035; 2023-04-01 14:56:24,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-01 14:56:24,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-01 14:56:24,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680360984509, max timestamp=1680360984521Num uniques 1035; 2023-04-01 14:56:24,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-01 14:56:24,524 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,524 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,525 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,525 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:24,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:24,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:24,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:24,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:24,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:24,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:24,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:24,526 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,526 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:24,526 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:24,526 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:24,526 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:24,527 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:24,528 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,528 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,529 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,529 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:24,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bbb82b3[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:24,531 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:24,531 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,532 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,532 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:24,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:24,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:24,533 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:24,534 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,534 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:24,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:24,538 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:24,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:24,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:24,538 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:24,538 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:24,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680360984535, max timestamp=1680360984541Num uniques 1094; 2023-04-01 14:56:24,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-01 14:56:24,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680360984535, max timestamp=1680360984541Num uniques 1094; 2023-04-01 14:56:24,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-01 14:56:24,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680360984535, max timestamp=1680360984541Num uniques 1094; 2023-04-01 14:56:24,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-01 14:56:24,544 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,546 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,548 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,549 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,549 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,549 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,549 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,549 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,549 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:24,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:24,549 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,549 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,549 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,549 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:24,550 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:24,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361044561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361044562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361044562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1680360984559, max timestamp=1680360984565Num uniques 1103; 2023-04-01 14:56:24,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1680360984559, max timestamp=1680360984565Num uniques 1103; 2023-04-01 14:56:24,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-01 14:56:24,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1680360984559, max timestamp=1680360984565Num uniques 1103; 2023-04-01 14:56:24,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-01 14:56:24,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-01 14:56:24,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680360984561, max timestamp=1680360984608Num uniques 1012; 2023-04-01 14:56:24,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-01 14:56:24,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680360984561, max timestamp=1680360984608Num uniques 1012; 2023-04-01 14:56:24,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-01 14:56:24,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680360984561, max timestamp=1680360984608Num uniques 1012; 2023-04-01 14:56:24,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-01 14:56:24,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680360984629, max timestamp=1680360984642Num uniques 1021; 2023-04-01 14:56:24,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-01 14:56:24,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680360984629, max timestamp=1680360984642Num uniques 1021; 2023-04-01 14:56:24,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680360984629, max timestamp=1680360984642Num uniques 1021; 2023-04-01 14:56:24,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-01 14:56:24,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-01 14:56:24,650 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:24,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:24,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:24,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:24,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:24,653 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:24,653 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,653 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:24,653 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:24,654 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,654 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,655 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,655 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,656 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,656 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:24,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:24,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:24,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:24,657 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:24,657 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,658 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:24,658 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:24,658 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:24,658 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,658 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:24,658 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:24,658 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,658 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:24,658 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:24,658 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,658 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:24,658 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:24,659 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:24,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@613814ef[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:24,661 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:24,661 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,664 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:24,664 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,664 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:24,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,666 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,666 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:24,668 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:24,668 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:24,668 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:24,669 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:24,669 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:24,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,670 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,671 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,671 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,673 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680360984672, max timestamp=1680360984677Num uniques 1030; 2023-04-01 14:56:24,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-01 14:56:24,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680360984672, max timestamp=1680360984677Num uniques 1030; 2023-04-01 14:56:24,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-01 14:56:24,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680360984672, max timestamp=1680360984677Num uniques 1030; 2023-04-01 14:56:24,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-01 14:56:24,680 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,680 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,680 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,680 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,681 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,681 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:24,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:24,681 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:24,681 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,681 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,681 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,681 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:24,681 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:24,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1680360984691, max timestamp=1680360984698Num uniques 1089; 2023-04-01 14:56:24,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8646, totalHeapSize=19896, min timestamp=1680360984691, max timestamp=1680360984698Num uniques 1089; 2023-04-01 14:56:24,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49866, new segment data size=8646, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-01 14:56:24,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8644, totalHeapSize=19896, min timestamp=1680360984691, max timestamp=1680360984698Num uniques 1089; 2023-04-01 14:56:24,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-01 14:56:24,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49864, new segment data size=8644, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-01 14:56:24,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13638, totalHeapSize=31232, min timestamp=1680360984711, max timestamp=1680360984717Num uniques 1048; 2023-04-01 14:56:24,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47988, new segment data size=13638, suffix heap size=136960,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=298 2023-04-01 14:56:24,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1680360984711, max timestamp=1680360984717Num uniques 1048; 2023-04-01 14:56:24,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=13648, suffix heap size=136960,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=298 2023-04-01 14:56:24,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9058, totalHeapSize=20832, min timestamp=1680360984712, max timestamp=1680360984717Num uniques 1048; 2023-04-01 14:56:24,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47988, new segment data size=9058, suffix heap size=136960,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=198 2023-04-01 14:56:24,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361044739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1680360984735, max timestamp=1680360984744Num uniques 1157; 2023-04-01 14:56:24,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1680360984735, max timestamp=1680360984744Num uniques 1157; 2023-04-01 14:56:24,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=7185, suffix heap size=148296,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=157 2023-04-01 14:56:24,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1680360984735, max timestamp=1680360984744Num uniques 1057; 2023-04-01 14:56:24,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=7185, suffix heap size=148296,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=157 2023-04-01 14:56:24,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=7185, suffix heap size=137896,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=157 2023-04-01 14:56:24,781 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:24,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:24,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:24,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:24,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:24,785 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:24,785 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,785 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:24,785 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:24,785 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,785 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,786 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1680360984759, max timestamp=1680360984786Num uniques 1016; 2023-04-01 14:56:24,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=12176, suffix heap size=133632,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=266 2023-04-01 14:56:24,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1680360984759, max timestamp=1680360984786Num uniques 1016; 2023-04-01 14:56:24,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=12176, suffix heap size=133632,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=266 2023-04-01 14:56:24,787 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,792 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:24,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1680360984759, max timestamp=1680360984786Num uniques 1016; 2023-04-01 14:56:24,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=12176, suffix heap size=133632,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=266 2023-04-01 14:56:24,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:24,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:24,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,800 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:24,808 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:24,808 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:24,808 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:24,808 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:24,810 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,810 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,810 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,810 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:24,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705d62a6[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:24,811 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:24,812 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,813 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:24,816 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,816 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:24,818 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:24,818 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,818 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,818 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:24,821 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:24,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:24,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:24,821 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:24,821 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:24,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,825 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,826 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,829 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:24,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:24,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,829 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,829 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:24,829 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:24,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361044822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361044827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8000, totalHeapSize=18440, min timestamp=1680360984807, max timestamp=1680360984826Num uniques 1125; 2023-04-01 14:56:24,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51510, new segment data size=8000, suffix heap size=144968,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=175 2023-04-01 14:56:24,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8004, totalHeapSize=18440, min timestamp=1680360984807, max timestamp=1680360984826Num uniques 1125; 2023-04-01 14:56:24,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51514, new segment data size=8004, suffix heap size=144968,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=175 2023-04-01 14:56:24,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1680360984807, max timestamp=1680360984822Num uniques 1125; 2023-04-01 14:56:24,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=8010, suffix heap size=144968,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=175 2023-04-01 14:56:24,835 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=921, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/dee770618b204057a19ad0f4969401da 2023-04-01 14:56:24,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=921 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/2d50aa7c94094073a30f00e194edcc5f 2023-04-01 14:56:24,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=13004, totalHeapSize=29776, min timestamp=1680360984840, max timestamp=1680360984874Num uniques 1034; 2023-04-01 14:56:24,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=13004, suffix heap size=135504,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=284 2023-04-01 14:56:24,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=13004, totalHeapSize=29776, min timestamp=1680360984840, max timestamp=1680360984874Num uniques 1034; 2023-04-01 14:56:24,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=13004, suffix heap size=135504,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=284 2023-04-01 14:56:24,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680360984840, max timestamp=1680360984874Num uniques 1034; 2023-04-01 14:56:24,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-04-01 14:56:24,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=921 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/653293c34ea24e9f83a10de42fe3a519 2023-04-01 14:56:24,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/dee770618b204057a19ad0f4969401da as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/dee770618b204057a19ad0f4969401da 2023-04-01 14:56:24,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/dee770618b204057a19ad0f4969401da, entries=150, sequenceid=921, filesize=25.4 K 2023-04-01 14:56:24,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/2d50aa7c94094073a30f00e194edcc5f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2d50aa7c94094073a30f00e194edcc5f 2023-04-01 14:56:24,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2d50aa7c94094073a30f00e194edcc5f, entries=150, sequenceid=921, filesize=11.9 K 2023-04-01 14:56:24,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/653293c34ea24e9f83a10de42fe3a519 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/653293c34ea24e9f83a10de42fe3a519 2023-04-01 14:56:24,902 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/653293c34ea24e9f83a10de42fe3a519, entries=150, sequenceid=921, filesize=11.9 K 2023-04-01 14:56:24,902 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=98.40 KB/100760 for d5537299121eaceb1e94c26142bdabf4 in 885ms, sequenceid=921, compaction requested=true 2023-04-01 14:56:24,903 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:24,903 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:24,903 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:56:24,903 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:56:24,903 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:24,903 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:24,904 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137986 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:56:24,904 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/A is initiating minor compaction (all files) 2023-04-01 14:56:24,905 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/A in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,905 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/38aaab2f28ea41369ce7ad0c58848591, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7b60d84e122d47babc4781d7084900e6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/122433d501ce467f88d048808135f51f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/12ba21aafd1c4b8b80ba98ca0ebd0508, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/dee770618b204057a19ad0f4969401da] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=134.8 K 2023-04-01 14:56:24,905 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:56:24,905 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/B is initiating minor compaction (all files) 2023-04-01 14:56:24,905 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/B in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,905 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e522c99371c84d7288463e3d6381e2c9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/58904f78b40c4dce9dbe0c6527711c8e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/f6aff69c537947ceb37901d7d9ed1c5c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d05248fd7ba740de94bd43e47eda8728, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2d50aa7c94094073a30f00e194edcc5f] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=60.2 K 2023-04-01 14:56:24,905 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 38aaab2f28ea41369ce7ad0c58848591, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680360980358 2023-04-01 14:56:24,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:24,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=105.11 KB heapSize=288.50 KB 2023-04-01 14:56:24,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:24,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:24,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:24,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:24,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:24,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:24,908 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting e522c99371c84d7288463e3d6381e2c9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680360980358 2023-04-01 14:56:24,908 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 7b60d84e122d47babc4781d7084900e6, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=851, earliestPutTs=1680360981853 2023-04-01 14:56:24,908 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 58904f78b40c4dce9dbe0c6527711c8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=851, earliestPutTs=1680360981853 2023-04-01 14:56:24,909 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 122433d501ce467f88d048808135f51f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1680360981907 2023-04-01 14:56:24,909 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f6aff69c537947ceb37901d7d9ed1c5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1680360981907 2023-04-01 14:56:24,909 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 12ba21aafd1c4b8b80ba98ca0ebd0508, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=900, earliestPutTs=1680360982508 2023-04-01 14:56:24,909 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting d05248fd7ba740de94bd43e47eda8728, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=900, earliestPutTs=1680360982508 2023-04-01 14:56:24,910 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting dee770618b204057a19ad0f4969401da, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=921, earliestPutTs=1680360983691 2023-04-01 14:56:24,910 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2d50aa7c94094073a30f00e194edcc5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=921, earliestPutTs=1680360983691 2023-04-01 14:56:24,925 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#B#compaction#634 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:24,929 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:24,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:24,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:24,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361044929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361044930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361044934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361044935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:24,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:24,942 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:24,943 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:24,943 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:24,943 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:24,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361044939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,944 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,944 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,946 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#A#compaction#633 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:24,947 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,948 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:24,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:24,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:24,964 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,964 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:24,964 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:24,964 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,964 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:24,964 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:24,964 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,964 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:24,964 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:24,967 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,967 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,968 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401223f0f5f9f0c4e85ac4585718c916e43 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401223f0f5f9f0c4e85ac4585718c916e43 2023-04-01 14:56:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,969 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,970 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/310fca4baa834bfbb5575fe6dba29b7c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/310fca4baa834bfbb5575fe6dba29b7c 2023-04-01 14:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b8e9b1e[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:24,970 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:24,970 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,971 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,971 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:24,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,972 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:24,973 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,973 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:24,973 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,973 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,975 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/B of d5537299121eaceb1e94c26142bdabf4 into 310fca4baa834bfbb5575fe6dba29b7c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:24,975 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:24,975 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/B, priority=11, startTime=1680360984903; duration=0sec 2023-04-01 14:56:24,976 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:24,976 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-01 14:56:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:24,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:24,985 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:24,985 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:24,985 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:24,985 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:24,985 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:24,986 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-01 14:56:24,986 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/C is initiating minor compaction (all files) 2023-04-01 14:56:24,986 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/C in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:24,986 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/ff1bb16f54b04e73b927ccd9b724ca2f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/3872819f04a54aecb82e37a5928d4659, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/a634f1afffb142cf8ac5607b30fa43a3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/81df411ac7c94ccaa6a53486e02fed68, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/653293c34ea24e9f83a10de42fe3a519] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=60.2 K 2023-04-01 14:56:24,987 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting ff1bb16f54b04e73b927ccd9b724ca2f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680360980358 2023-04-01 14:56:24,987 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 3872819f04a54aecb82e37a5928d4659, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=851, earliestPutTs=1680360981853 2023-04-01 14:56:24,987 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting a634f1afffb142cf8ac5607b30fa43a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1680360981907 2023-04-01 14:56:24,987 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,987 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,988 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:24,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,989 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 81df411ac7c94ccaa6a53486e02fed68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=900, earliestPutTs=1680360982508 2023-04-01 14:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,989 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,989 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 653293c34ea24e9f83a10de42fe3a519, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=921, earliestPutTs=1680360983691 2023-04-01 14:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:24,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:24,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:24,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:24,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:24,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:24,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:24,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:24,995 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:24,996 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:24,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:24,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:24,996 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:24,996 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:24,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:24,999 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#C#compaction#635 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:25,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361045037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361045037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361045040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,096 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:25,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:25,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:25,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:25,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:25,099 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:25,099 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,099 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:25,099 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:25,100 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,100 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,101 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,102 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,102 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,102 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:25,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:25,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:25,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:25,103 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,103 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:25,103 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:25,103 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:25,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:25,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:25,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:25,105 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,105 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c6730c6[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:25,107 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:25,107 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,108 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:25,109 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,109 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:25,112 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:25,114 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:25,114 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:25,114 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:25,114 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:25,114 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:25,116 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,116 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,116 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,117 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,118 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,122 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,122 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,122 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,122 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,122 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,122 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,122 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:25,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:25,122 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,124 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:25,124 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:25,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361045140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361045144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,224 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:25,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:25,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:25,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:25,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:25,227 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:25,227 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,227 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:25,227 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:25,228 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,228 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,231 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,231 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,232 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,232 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:25,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:25,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:25,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:25,233 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,233 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:25,233 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:25,233 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:25,233 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:25,235 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,235 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,235 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,235 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dd9d4c9[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:25,237 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:25,237 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,238 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,238 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:25,238 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:25,238 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:25,238 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,238 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:25,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 424 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361045241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361045241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 461 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361045242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:25,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:25,250 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:25,250 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:25,250 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:25,250 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:25,250 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:25,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,253 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,254 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,256 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,256 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,256 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,256 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:25,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:25,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:25,257 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,257 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,257 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,257 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,257 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,257 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,257 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:25,257 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:25,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,357 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:25,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:25,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:25,360 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:25,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:25,360 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:25,361 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,361 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,362 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,362 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,362 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,362 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:25,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:25,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:25,364 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,364 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:25,364 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:25,364 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:25,366 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,366 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,366 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,366 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60add86[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:25,367 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:25,367 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,369 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:25,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:25,369 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:25,369 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,369 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,370 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:25,373 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:25,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:25,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:25,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:25,373 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:25,375 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,375 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,375 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,376 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,377 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,378 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:25,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:25,378 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,379 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:25,379 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:25,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,379 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,379 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:25,385 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1382, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/7686ce331ae740c5b83751f37275a5b7 2023-04-01 14:56:25,394 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/1a931aed88d746a7a0334ef576c3b2dd as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/1a931aed88d746a7a0334ef576c3b2dd 2023-04-01 14:56:25,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1382 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/1cc120fbce0944ab9b8678d171613cbb 2023-04-01 14:56:25,399 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/A of d5537299121eaceb1e94c26142bdabf4 into 1a931aed88d746a7a0334ef576c3b2dd(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:25,399 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:25,399 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/A, priority=11, startTime=1680360984903; duration=0sec 2023-04-01 14:56:25,399 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:25,409 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/c3758a3e935642e29f8a2f5304921fe5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/c3758a3e935642e29f8a2f5304921fe5 2023-04-01 14:56:25,413 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/C of d5537299121eaceb1e94c26142bdabf4 into c3758a3e935642e29f8a2f5304921fe5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:25,413 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:25,413 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/C, priority=11, startTime=1680360984903; duration=0sec 2023-04-01 14:56:25,413 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:25,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361045447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361045448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,479 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:25,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:25,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:25,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:25,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:25,482 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:25,482 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,482 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:25,482 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:25,483 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,483 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,486 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,486 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,486 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,486 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:25,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:25,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:25,487 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,487 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:25,488 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:25,488 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9d3e032[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:25,491 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:25,491 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,492 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:25,492 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,492 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:25,493 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,493 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:25,493 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,493 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:25,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:25,496 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:25,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:25,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:25,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:25,496 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:25,501 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,501 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,501 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,502 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,505 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:25,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:25,505 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,506 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:25,506 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:25,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361045545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361045545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:25,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361045548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,606 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:25,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:25,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:25,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:25,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:25,610 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:25,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:25,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:25,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,610 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,611 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,612 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,612 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:25,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:25,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:25,613 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:25,613 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,613 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:25,614 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:25,614 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:25,614 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,614 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:25,615 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:25,615 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:25,615 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:25,619 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:25,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:25,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cb710ce[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:25,621 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:25,621 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,624 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:25,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,624 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:25,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:25,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,625 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:25,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:25,627 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:25,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:25,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:25,628 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:25,628 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:25,629 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,629 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,629 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,630 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,631 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,633 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,633 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:25,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:25,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,634 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:25,634 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:25,633 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,633 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:25,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,734 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:25,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:25,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:25,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:25,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:25,737 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:25,737 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,737 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:25,737 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:25,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,738 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,740 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,740 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:25,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:25,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:25,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:25,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:25,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:25,741 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:25,742 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,742 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:25,742 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:25,742 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:25,742 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,742 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:25,742 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:25,742 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,742 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:25,742 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:25,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,743 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,743 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:25,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:25,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f9bf20d[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:25,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:25,746 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,747 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,747 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:25,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,747 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:25,747 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:25,747 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,748 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:25,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:25,754 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:25,754 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:25,754 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:25,754 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:25,754 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:25,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,759 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,760 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,761 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,763 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:25,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:25,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,764 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,764 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,764 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,764 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,764 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:25,764 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:25,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,805 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1382 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/f9bd86836a534b1bb7f81a7a57ddca01 2023-04-01 14:56:25,808 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/7686ce331ae740c5b83751f37275a5b7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7686ce331ae740c5b83751f37275a5b7 2023-04-01 14:56:25,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7686ce331ae740c5b83751f37275a5b7, entries=250, sequenceid=1382, filesize=39.2 K 2023-04-01 14:56:25,813 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/1cc120fbce0944ab9b8678d171613cbb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1cc120fbce0944ab9b8678d171613cbb 2023-04-01 14:56:25,816 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1cc120fbce0944ab9b8678d171613cbb, entries=150, sequenceid=1382, filesize=11.9 K 2023-04-01 14:56:25,816 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/f9bd86836a534b1bb7f81a7a57ddca01 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f9bd86836a534b1bb7f81a7a57ddca01 2023-04-01 14:56:25,819 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f9bd86836a534b1bb7f81a7a57ddca01, entries=150, sequenceid=1382, filesize=11.9 K 2023-04-01 14:56:25,820 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~111.82 KB/114500, heapSize ~308.38 KB/315776, currentSize=73.80 KB/75570 for d5537299121eaceb1e94c26142bdabf4 in 914ms, sequenceid=1382, compaction requested=false 2023-04-01 14:56:25,820 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:25,864 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:25,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:25,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:25,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:25,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:25,878 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:25,878 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,878 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:25,878 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:25,879 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,879 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,880 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:25,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,880 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:25,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:25,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:25,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:25,881 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:25,881 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:25,881 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:25,881 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:25,882 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:25,882 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,882 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:25,882 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:56:25,882 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:25,882 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:25,882 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:25,882 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:25,882 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:25,882 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:25,905 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040151bf9679df2e405593656865310b1414 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040151bf9679df2e405593656865310b1414 2023-04-01 14:56:25,913 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1399, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/6cfdc6cda95c47a3919ef4f9d53e8371 2023-04-01 14:56:25,937 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1399 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/2299b55e9e6948d79a8a4f803f61932c 2023-04-01 14:56:25,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:25,953 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:25,963 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1399 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/cd55fe3203d948038577842cb4121efe 2023-04-01 14:56:25,967 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/6cfdc6cda95c47a3919ef4f9d53e8371 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/6cfdc6cda95c47a3919ef4f9d53e8371 2023-04-01 14:56:25,971 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/6cfdc6cda95c47a3919ef4f9d53e8371, entries=150, sequenceid=1399, filesize=25.4 K 2023-04-01 14:56:25,972 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/2299b55e9e6948d79a8a4f803f61932c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2299b55e9e6948d79a8a4f803f61932c 2023-04-01 14:56:25,975 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2299b55e9e6948d79a8a4f803f61932c, entries=150, sequenceid=1399, filesize=11.9 K 2023-04-01 14:56:25,976 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/cd55fe3203d948038577842cb4121efe as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/cd55fe3203d948038577842cb4121efe 2023-04-01 14:56:25,979 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/cd55fe3203d948038577842cb4121efe, entries=150, sequenceid=1399, filesize=11.9 K 2023-04-01 14:56:25,979 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d5537299121eaceb1e94c26142bdabf4 in 97ms, sequenceid=1399, compaction requested=true 2023-04-01 14:56:25,979 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:25,979 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:25,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:25,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:25,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:25,979 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:25,981 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,981 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,981 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,981 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:25,982 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:56:25,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:25,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:25,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:25,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:25,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:25,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:25,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:25,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f2df295[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:25,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:25,983 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,985 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:25,986 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,986 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:25,987 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,987 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,987 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:25,991 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:25,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:25,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:25,991 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:25,991 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:25,994 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,994 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:25,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,994 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:25,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:25,995 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040148d430f0eb8647ac9d25288f0c6e1c88 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040148d430f0eb8647ac9d25288f0c6e1c88 2023-04-01 14:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:25,995 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,998 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,001 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1418, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/b79e2c00b20e4e83be7d2844716ee003 2023-04-01 14:56:26,001 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,001 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,001 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,001 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,001 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,001 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,001 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:26,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361046000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361046000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,003 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:26,003 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:26,003 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:26,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:26,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:26,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:26,004 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:26,004 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:26,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:26,011 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1418 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/6b1b5e4221134d7f9eab0b9cbdf7cef3 2023-04-01 14:56:26,039 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1418 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/2af6349daab54f0fa2b9c2868de4a3b6 2023-04-01 14:56:26,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/b79e2c00b20e4e83be7d2844716ee003 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/b79e2c00b20e4e83be7d2844716ee003 2023-04-01 14:56:26,047 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/b79e2c00b20e4e83be7d2844716ee003, entries=150, sequenceid=1418, filesize=25.4 K 2023-04-01 14:56:26,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/6b1b5e4221134d7f9eab0b9cbdf7cef3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/6b1b5e4221134d7f9eab0b9cbdf7cef3 2023-04-01 14:56:26,052 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/6b1b5e4221134d7f9eab0b9cbdf7cef3, entries=150, sequenceid=1418, filesize=11.9 K 2023-04-01 14:56:26,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361046048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,053 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/2af6349daab54f0fa2b9c2868de4a3b6 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2af6349daab54f0fa2b9c2868de4a3b6 2023-04-01 14:56:26,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361046050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,060 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2af6349daab54f0fa2b9c2868de4a3b6, entries=150, sequenceid=1418, filesize=11.9 K 2023-04-01 14:56:26,061 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d5537299121eaceb1e94c26142bdabf4 in 79ms, sequenceid=1418, compaction requested=true 2023-04-01 14:56:26,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:26,061 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:26,061 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:26,061 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:56:26,061 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:56:26,061 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-01 14:56:26,063 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 119154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:56:26,063 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/A is initiating minor compaction (all files) 2023-04-01 14:56:26,063 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/A in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,063 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/1a931aed88d746a7a0334ef576c3b2dd, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7686ce331ae740c5b83751f37275a5b7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/6cfdc6cda95c47a3919ef4f9d53e8371, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/b79e2c00b20e4e83be7d2844716ee003] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=116.4 K 2023-04-01 14:56:26,063 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:56:26,063 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 1a931aed88d746a7a0334ef576c3b2dd, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=921, earliestPutTs=1680360983691 2023-04-01 14:56:26,063 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/B is initiating minor compaction (all files) 2023-04-01 14:56:26,064 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/B in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,064 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/310fca4baa834bfbb5575fe6dba29b7c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1cc120fbce0944ab9b8678d171613cbb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2299b55e9e6948d79a8a4f803f61932c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/6b1b5e4221134d7f9eab0b9cbdf7cef3] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=48.5 K 2023-04-01 14:56:26,064 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 7686ce331ae740c5b83751f37275a5b7, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=1382, earliestPutTs=1680360984890 2023-04-01 14:56:26,064 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 310fca4baa834bfbb5575fe6dba29b7c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=921, earliestPutTs=1680360983691 2023-04-01 14:56:26,065 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting 6cfdc6cda95c47a3919ef4f9d53e8371, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1399, earliestPutTs=1680360984925 2023-04-01 14:56:26,065 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 1cc120fbce0944ab9b8678d171613cbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1382, earliestPutTs=1680360984896 2023-04-01 14:56:26,065 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] compactions.Compactor(207): Compacting b79e2c00b20e4e83be7d2844716ee003, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1418, earliestPutTs=1680360985976 2023-04-01 14:56:26,065 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2299b55e9e6948d79a8a4f803f61932c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1399, earliestPutTs=1680360984925 2023-04-01 14:56:26,066 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 6b1b5e4221134d7f9eab0b9cbdf7cef3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1418, earliestPutTs=1680360985976 2023-04-01 14:56:26,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:26,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:56:26,067 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:26,067 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:26,067 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:26,067 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:26,067 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:26,067 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:26,089 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#B#compaction#646 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:26,093 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#A#compaction#644 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:26,104 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:26,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:26,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:26,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361046111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361046112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361046116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:26,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:26,121 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:26,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:26,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:26,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:26,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,121 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:26,123 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,124 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,124 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:26,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:26,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:26,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:26,125 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,125 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:26,125 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:26,125 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:26,125 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,126 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:26,126 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:26,127 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,127 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,127 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:26,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e95132[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:26,129 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:26,129 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,133 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:26,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:26,133 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:26,133 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,133 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:26,134 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:26,134 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:26,134 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:26,134 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:26,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:26,137 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:26,137 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:26,137 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:26,137 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:26,137 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:26,138 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,138 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,138 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:26,139 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,140 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,142 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,142 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:26,142 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,142 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:26,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:26,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:26,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:26,143 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:26,143 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:26,143 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:26,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:26,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361046217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361046217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361046221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,243 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:26,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:26,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:26,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:26,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:26,246 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:26,246 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:26,246 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:26,246 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:26,247 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,247 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,248 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,248 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:26,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:26,248 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,248 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:26,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:26,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:26,250 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,250 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:26,250 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:26,250 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:26,252 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,252 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,252 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,252 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:26,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d6c03a1[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:26,254 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:26,254 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,255 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,255 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:26,255 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:26,255 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:26,255 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,255 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:26,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:26,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:26,260 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:26,260 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:26,260 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:26,260 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:26,260 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:26,262 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,262 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,262 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,263 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,264 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:26,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:26,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:26,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:26,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:26,267 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:26,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,267 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:26,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:26,268 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:26,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:26,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:26,368 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:26,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:26,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:26,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:26,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:26,371 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:26,371 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:26,371 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:26,371 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:26,371 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,371 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,372 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,372 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:26,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:26,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,373 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:26,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:26,375 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,375 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:26,375 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:26,375 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:26,377 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,378 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,378 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,378 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de7879b[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:26,379 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:26,379 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,380 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:26,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:26,380 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:26,380 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,381 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:26,381 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,381 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:26,382 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:26,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,382 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:26,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:26,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:26,384 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:26,384 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:26,384 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:26,384 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:26,384 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:26,385 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,385 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,385 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,386 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,387 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,391 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,391 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:26,391 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,391 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:26,391 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,391 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:26,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:26,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:26,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:26,391 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,392 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,392 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,392 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,392 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:26,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:26,392 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:26,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361046421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361046421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 481 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361046425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,492 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:26,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:26,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:26,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:26,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:26,496 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:26,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:26,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:26,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:26,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,496 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,498 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:26,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,499 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:26,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:26,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:26,500 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,500 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,501 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:26,501 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:26,501 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:26,501 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:26,502 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304011902e073df88467f9d81ae40efb3fe3c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011902e073df88467f9d81ae40efb3fe3c 2023-04-01 14:56:26,504 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,504 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,505 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,505 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:26,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3e61e8[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:26,506 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:26,506 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,508 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1432, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/e872bde770004631a241621b641033cb 2023-04-01 14:56:26,508 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:26,508 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:26,508 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,508 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:26,508 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:26,513 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,513 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:26,514 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,514 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,514 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/acf83307a17e456caf64f46ccb78de27 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/acf83307a17e456caf64f46ccb78de27 2023-04-01 14:56:26,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:26,518 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:26,518 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:26,518 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:26,518 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:26,518 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:26,520 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,521 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,521 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,521 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:26,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,522 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/B of d5537299121eaceb1e94c26142bdabf4 into acf83307a17e456caf64f46ccb78de27(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:26,522 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:26,522 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/B, priority=12, startTime=1680360986061; duration=0sec 2023-04-01 14:56:26,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:26,522 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-01 14:56:26,522 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-01 14:56:26,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:26,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:26,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1432 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/923f283b6f1848ae9a40fcfe677296f5 2023-04-01 14:56:26,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,524 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,526 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-01 14:56:26,527 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1912): d5537299121eaceb1e94c26142bdabf4/C is initiating minor compaction (all files) 2023-04-01 14:56:26,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,527 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d5537299121eaceb1e94c26142bdabf4/C in TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:26,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:26,527 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:26,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:26,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,527 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:26,527 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:26,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:26,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:26,527 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/c3758a3e935642e29f8a2f5304921fe5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f9bd86836a534b1bb7f81a7a57ddca01, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/cd55fe3203d948038577842cb4121efe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2af6349daab54f0fa2b9c2868de4a3b6] into tmpdir=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp, totalSize=48.5 K 2023-04-01 14:56:26,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,528 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,528 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:26,528 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:26,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:26,528 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting c3758a3e935642e29f8a2f5304921fe5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=921, earliestPutTs=1680360983691 2023-04-01 14:56:26,529 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting f9bd86836a534b1bb7f81a7a57ddca01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1382, earliestPutTs=1680360984896 2023-04-01 14:56:26,530 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting cd55fe3203d948038577842cb4121efe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1399, earliestPutTs=1680360984925 2023-04-01 14:56:26,530 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] compactions.Compactor(207): Compacting 2af6349daab54f0fa2b9c2868de4a3b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1418, earliestPutTs=1680360985976 2023-04-01 14:56:26,544 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1432 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/9058501263064e11869a21dc7cf8bd64 2023-04-01 14:56:26,546 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] throttle.PressureAwareThroughputController(145): d5537299121eaceb1e94c26142bdabf4#C#compaction#649 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-01 14:56:26,558 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/e872bde770004631a241621b641033cb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e872bde770004631a241621b641033cb 2023-04-01 14:56:26,559 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/df6dfcf5750e4c26962897e4326b8d1d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/df6dfcf5750e4c26962897e4326b8d1d 2023-04-01 14:56:26,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e872bde770004631a241621b641033cb, entries=150, sequenceid=1432, filesize=25.4 K 2023-04-01 14:56:26,565 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/923f283b6f1848ae9a40fcfe677296f5 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/923f283b6f1848ae9a40fcfe677296f5 2023-04-01 14:56:26,570 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/923f283b6f1848ae9a40fcfe677296f5, entries=150, sequenceid=1432, filesize=11.9 K 2023-04-01 14:56:26,571 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/9058501263064e11869a21dc7cf8bd64 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/9058501263064e11869a21dc7cf8bd64 2023-04-01 14:56:26,572 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/A of d5537299121eaceb1e94c26142bdabf4 into df6dfcf5750e4c26962897e4326b8d1d(size=26.4 K), total size for store is 51.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:26,572 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:26,572 INFO [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/A, priority=12, startTime=1680360986061; duration=0sec 2023-04-01 14:56:26,572 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:26,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/9058501263064e11869a21dc7cf8bd64, entries=150, sequenceid=1432, filesize=11.9 K 2023-04-01 14:56:26,576 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/09ccf2544517411bb721f66c7400b627 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/09ccf2544517411bb721f66c7400b627 2023-04-01 14:56:26,576 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d5537299121eaceb1e94c26142bdabf4 in 509ms, sequenceid=1432, compaction requested=false 2023-04-01 14:56:26,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:26,579 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d5537299121eaceb1e94c26142bdabf4/C of d5537299121eaceb1e94c26142bdabf4 into 09ccf2544517411bb721f66c7400b627(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-01 14:56:26,579 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:26,579 INFO [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4., storeName=d5537299121eaceb1e94c26142bdabf4/C, priority=12, startTime=1680360986061; duration=0sec 2023-04-01 14:56:26,579 DEBUG [RS:0;jenkins-hbase4:40863-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:26,627 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:26,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:26,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:26,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:26,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:26,631 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:26,631 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:26,631 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:26,631 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:26,631 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,631 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,634 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:26,634 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:26,634 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,634 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:26,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:26,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:26,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:26,636 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:26,636 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:26,636 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:26,636 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:26,636 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:26,636 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:26,636 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:26,636 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-01 14:56:26,637 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:26,637 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:26,637 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:26,637 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:26,637 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:26,637 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:26,653 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401e843cc1df392489eb41a1905c07badc0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e843cc1df392489eb41a1905c07badc0 2023-04-01 14:56:26,666 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1453, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/274e55b3bf174533958b0ca0b18bb16f 2023-04-01 14:56:26,677 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1453 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/0fcb1580f0d14d9cbcc558690bfe3758 2023-04-01 14:56:26,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:26,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:26,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361046757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361046761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361046762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361046863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361046867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:26,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:26,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 488 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361046867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:27,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 430 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680361047053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41776 deadline: 1680361047063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:27,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361047066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:27,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361047070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:27,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 490 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361047070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,086 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1453 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/fb1f381c17a04d6c88d857a30c16eec7 2023-04-01 14:56:27,090 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/274e55b3bf174533958b0ca0b18bb16f as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/274e55b3bf174533958b0ca0b18bb16f 2023-04-01 14:56:27,093 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/274e55b3bf174533958b0ca0b18bb16f, entries=150, sequenceid=1453, filesize=25.4 K 2023-04-01 14:56:27,094 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/0fcb1580f0d14d9cbcc558690bfe3758 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/0fcb1580f0d14d9cbcc558690bfe3758 2023-04-01 14:56:27,097 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/0fcb1580f0d14d9cbcc558690bfe3758, entries=150, sequenceid=1453, filesize=11.9 K 2023-04-01 14:56:27,098 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/fb1f381c17a04d6c88d857a30c16eec7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/fb1f381c17a04d6c88d857a30c16eec7 2023-04-01 14:56:27,101 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/fb1f381c17a04d6c88d857a30c16eec7, entries=150, sequenceid=1453, filesize=11.9 K 2023-04-01 14:56:27,101 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d5537299121eaceb1e94c26142bdabf4 in 465ms, sequenceid=1453, compaction requested=true 2023-04-01 14:56:27,101 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:27,101 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:27,101 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,102 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:27,102 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:27,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:27,103 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,103 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:27,104 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:27,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:27,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e0dc9bc[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:27,105 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:27,105 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,106 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,107 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:27,107 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,107 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:27,107 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:27,107 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:27,109 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:27,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:27,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:27,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:27,109 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:27,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:27,111 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:27,112 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:27,112 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:27,112 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:27,112 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:27,116 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,116 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:27,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,117 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:27,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,118 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,119 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,122 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:27,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:27,122 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:27,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:27,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,123 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:27,122 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:27,123 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:27,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:27,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:27,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,123 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:27,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:27,223 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:27,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:27,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-01 14:56:27,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-01 14:56:27,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-01 14:56:27,228 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-01 14:56:27,228 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:27,228 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-01 14:56:27,228 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-01 14:56:27,229 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,229 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:27,230 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,230 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:27,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:27,230 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,230 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-01 14:56:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:27,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-01 14:56:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:27,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-01 14:56:27,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-01 14:56:27,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-01 14:56:27,231 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-01 14:56:27,231 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-01 14:56:27,232 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-01 14:56:27,232 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-01 14:56:27,232 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-01 14:56:27,232 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:27,232 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. started... 2023-04-01 14:56:27,232 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-01 14:56:27,232 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:27,232 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:27,232 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:27,232 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:27,233 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:27,233 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:27,244 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230401afaa8ca0a3a54a8f9079f786f91efcef to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401afaa8ca0a3a54a8f9079f786f91efcef 2023-04-01 14:56:27,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:27,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. as already flushing 2023-04-01 14:56:27,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:27,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 496 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361047390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:27,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361047392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:27,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361047393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:27,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40863] ipc.CallRunner(144): callId: 498 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41784 deadline: 1680361047494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:27,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40863] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41794 deadline: 1680361047495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-01 14:56:27,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41788 deadline: 1680361047496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,649 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1467, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/b59e2d636c974c08817d036d62e21cc9 2023-04-01 14:56:27,660 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1467 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/1aa2b8dc9c8842ab8bc3b3e7c4f99352 2023-04-01 14:56:27,669 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1467 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/4d3f05b7777344648cb434c33a73aaa3 2023-04-01 14:56:27,672 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/b59e2d636c974c08817d036d62e21cc9 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/b59e2d636c974c08817d036d62e21cc9 2023-04-01 14:56:27,673 DEBUG [Thread-4040] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x618c0022 to 127.0.0.1:60465 2023-04-01 14:56:27,673 DEBUG [Thread-4040] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:27,674 DEBUG [Thread-4032] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:60465 2023-04-01 14:56:27,674 DEBUG [Thread-4032] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:27,675 DEBUG [Thread-4036] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fae3ed2 to 127.0.0.1:60465 2023-04-01 14:56:27,675 DEBUG [Thread-4036] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:27,676 DEBUG [Thread-4034] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:60465 2023-04-01 14:56:27,676 DEBUG [Thread-4034] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:27,677 DEBUG [Thread-4038] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eedd898 to 127.0.0.1:60465 2023-04-01 14:56:27,677 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/b59e2d636c974c08817d036d62e21cc9, entries=150, sequenceid=1467, filesize=25.4 K 2023-04-01 14:56:27,677 DEBUG [Thread-4038] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:27,677 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/1aa2b8dc9c8842ab8bc3b3e7c4f99352 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1aa2b8dc9c8842ab8bc3b3e7c4f99352 2023-04-01 14:56:27,681 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1aa2b8dc9c8842ab8bc3b3e7c4f99352, entries=150, sequenceid=1467, filesize=11.9 K 2023-04-01 14:56:27,681 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/4d3f05b7777344648cb434c33a73aaa3 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/4d3f05b7777344648cb434c33a73aaa3 2023-04-01 14:56:27,684 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/4d3f05b7777344648cb434c33a73aaa3, entries=150, sequenceid=1467, filesize=11.9 K 2023-04-01 14:56:27,684 INFO [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d5537299121eaceb1e94c26142bdabf4 in 452ms, sequenceid=1467, compaction requested=true 2023-04-01 14:56:27,684 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:27,684 DEBUG [rs(jenkins-hbase4.apache.org,40863,1680360818567)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:27,684 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-01 14:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-01 14:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-01 14:56:27,685 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-01 14:56:27,686 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,686 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:27,686 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,686 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-01 14:56:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40863,1680360818567' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-01 14:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53b1b0d7[Count = 0] remaining members to acquire global barrier 2023-04-01 14:56:27,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-01 14:56:27,687 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,688 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,688 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-01 14:56:27,689 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,689 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-01 14:56:27,689 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-01 14:56:27,689 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40863,1680360818567' in zk 2023-04-01 14:56:27,691 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-01 14:56:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:27,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:27,691 DEBUG [member: 'jenkins-hbase4.apache.org,40863,1680360818567' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-01 14:56:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40863,1680360818567': 2023-04-01 14:56:27,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40863,1680360818567' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-01 14:56:27,693 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-01 14:56:27,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-01 14:56:27,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-01 14:56:27,693 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-01 14:56:27,693 INFO [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-01 14:56:27,694 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,695 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,695 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-01 14:56:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-01 14:56:27,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:27,695 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-01 14:56:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-01 14:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,696 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-01 14:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-01 14:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-01 14:56:27,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-01 14:56:27,698 DEBUG [(jenkins-hbase4.apache.org,35215,1680360817296)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-01 14:56:27,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-01 14:56:27,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-01 14:56:27,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-01 14:56:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-01 14:56:27,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:27,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-01 14:56:27,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-01 14:56:27,698 DEBUG [Thread-4031] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-01 14:56:27,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:27,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-01 14:56:27,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-01 14:56:27,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-01 14:56:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-01 14:56:27,698 DEBUG [Thread-4031] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-01 14:56:27,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40863] regionserver.HRegion(9158): Flush requested on d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:27,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-01 14:56:27,699 DEBUG [Thread-4021] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:60465 2023-04-01 14:56:27,699 DEBUG [Thread-4021] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:27,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:27,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:27,701 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:27,701 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:27,701 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:27,701 DEBUG [Thread-4029] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e6a6f3c to 127.0.0.1:60465 2023-04-01 14:56:27,701 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-01 14:56:27,701 DEBUG [Thread-4029] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:27,701 DEBUG [Thread-4023] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:60465 2023-04-01 14:56:27,701 DEBUG [Thread-4023] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:27,799 DEBUG [Thread-4031] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-01 14:56:27,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-01 14:56:28,111 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304015df83c7035c54a9cb4d8d47ea7a4f579 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015df83c7035c54a9cb4d8d47ea7a4f579 2023-04-01 14:56:28,515 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1488, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/75b796e694ac4d5595ecfec5fdb18e0b 2023-04-01 14:56:28,924 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1488 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/73f8a8b1b7df4a0fa40dc4ee3a6f3e5d 2023-04-01 14:56:29,068 DEBUG [Thread-4027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:60465 2023-04-01 14:56:29,068 DEBUG [Thread-4027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:29,075 DEBUG [Thread-4025] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:60465 2023-04-01 14:56:29,076 DEBUG [Thread-4025] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:29,076 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-01 14:56:29,076 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 349 2023-04-01 14:56:29,076 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 255 2023-04-01 14:56:29,076 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 240 2023-04-01 14:56:29,076 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 301 2023-04-01 14:56:29,076 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(394): wrote 236 2023-04-01 14:56:29,076 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-01 14:56:29,076 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4700 2023-04-01 14:56:29,076 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4586 2023-04-01 14:56:29,076 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4687 2023-04-01 14:56:29,076 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4703 2023-04-01 14:56:29,077 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(398): read 4614 2023-04-01 14:56:29,077 INFO [Listener at localhost/46667] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-01 14:56:29,077 INFO [Listener at localhost/46667] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 14:56:29,077 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:60465 2023-04-01 14:56:29,077 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:29,077 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-01 14:56:29,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-01 14:56:29,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:29,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-01 14:56:29,081 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360989080"}]},"ts":"1680360989080"} 2023-04-01 14:56:29,081 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-01 14:56:29,083 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-01 14:56:29,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d5537299121eaceb1e94c26142bdabf4, UNASSIGN}] 2023-04-01 14:56:29,084 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d5537299121eaceb1e94c26142bdabf4, UNASSIGN 2023-04-01 14:56:29,085 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=d5537299121eaceb1e94c26142bdabf4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:29,085 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360989085"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680360989085"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680360989085"}]},"ts":"1680360989085"} 2023-04-01 14:56:29,086 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567}] 2023-04-01 14:56:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-01 14:56:29,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:29,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d5537299121eaceb1e94c26142bdabf4, disabling compactions & flushes 2023-04-01 14:56:29,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:29,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1488 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/b9a1f781bb7a4c53a83fd35c5462e7c7 2023-04-01 14:56:29,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/75b796e694ac4d5595ecfec5fdb18e0b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/75b796e694ac4d5595ecfec5fdb18e0b 2023-04-01 14:56:29,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/75b796e694ac4d5595ecfec5fdb18e0b, entries=150, sequenceid=1488, filesize=25.4 K 2023-04-01 14:56:29,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/73f8a8b1b7df4a0fa40dc4ee3a6f3e5d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/73f8a8b1b7df4a0fa40dc4ee3a6f3e5d 2023-04-01 14:56:29,343 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/73f8a8b1b7df4a0fa40dc4ee3a6f3e5d, entries=150, sequenceid=1488, filesize=11.9 K 2023-04-01 14:56:29,344 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/b9a1f781bb7a4c53a83fd35c5462e7c7 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/b9a1f781bb7a4c53a83fd35c5462e7c7 2023-04-01 14:56:29,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/b9a1f781bb7a4c53a83fd35c5462e7c7, entries=150, sequenceid=1488, filesize=11.9 K 2023-04-01 14:56:29,347 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=13.42 KB/13740 for d5537299121eaceb1e94c26142bdabf4 in 1648ms, sequenceid=1488, compaction requested=true 2023-04-01 14:56:29,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:29,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:29,347 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. because compaction request was cancelled 2023-04-01 14:56:29,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:29,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:29,347 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. because compaction request was cancelled 2023-04-01 14:56:29,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:29,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. after waiting 0 ms 2023-04-01 14:56:29,347 DEBUG [RS:0;jenkins-hbase4:40863-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. because compaction request was cancelled 2023-04-01 14:56:29,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-01 14:56:29,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:29,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d5537299121eaceb1e94c26142bdabf4 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-01 14:56:29,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=A 2023-04-01 14:56:29,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:29,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=B 2023-04-01 14:56:29,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:29,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d5537299121eaceb1e94c26142bdabf4, store=C 2023-04-01 14:56:29,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-01 14:56:29,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-01 14:56:29,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-01 14:56:29,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304015aa480817d894af496360ecdb2249805 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015aa480817d894af496360ecdb2249805 2023-04-01 14:56:30,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1493, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/03d3af142da447058fcfb3e37fcbd668 2023-04-01 14:56:30,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1493 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/b8d581add7cc4f4e8bca5934ec392c01 2023-04-01 14:56:30,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-01 14:56:30,578 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1493 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/5f457a6f911543b187a4a23ba4171752 2023-04-01 14:56:30,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/A/03d3af142da447058fcfb3e37fcbd668 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/03d3af142da447058fcfb3e37fcbd668 2023-04-01 14:56:30,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/03d3af142da447058fcfb3e37fcbd668, entries=100, sequenceid=1493, filesize=18.6 K 2023-04-01 14:56:30,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/B/b8d581add7cc4f4e8bca5934ec392c01 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/b8d581add7cc4f4e8bca5934ec392c01 2023-04-01 14:56:30,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/b8d581add7cc4f4e8bca5934ec392c01, entries=100, sequenceid=1493, filesize=9.5 K 2023-04-01 14:56:30,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/.tmp/C/5f457a6f911543b187a4a23ba4171752 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/5f457a6f911543b187a4a23ba4171752 2023-04-01 14:56:30,590 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/5f457a6f911543b187a4a23ba4171752, entries=100, sequenceid=1493, filesize=9.5 K 2023-04-01 14:56:30,591 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for d5537299121eaceb1e94c26142bdabf4 in 1244ms, sequenceid=1493, compaction requested=true 2023-04-01 14:56:30,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5ccdc27eb0704b908d36e610535e8885, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d7c8e4d1229d4695a23bacf095693038, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e9bd859fff7b4ac99ec01c86d28c48bf, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d21d9f71e8a94901841f9416295defeb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/f0573b5473d04bdfb5de3dfe355f6c50, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/03b514f2737a4b4b94131ea0102085da, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/48aefbd1311b44e5ac98d956e27d7453, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/c9d6012403b34eafae22a867a7584630, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5f6a85c4651246769a6c9207bf565714, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/19385fa9f688465a9cc48493e2b5c0b7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e67e8805beb8478ea198391d847540f6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/cc33c609b12d456bb956de4c6d3c7579, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/a5f33c5ef15746d381f5d99cb7bdfd76, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/40e7339460f1480a8d7848eccc5e3d3e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/3c86eb3446954587a43b93259222bcc7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e914c686bf924e4c8509a8c7a2e8df31, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/915d3fa7b4cf422195db4a7a2c0f5549, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d190107168fb42f49dbb17b8bc6adfc9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/88fb8e21311a40a3a1e760cff5901908, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5b43a0a32d594a3ab0eac01fa48849da, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/38aaab2f28ea41369ce7ad0c58848591, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/ae917c1cb389464a90af32d762073e02, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7b60d84e122d47babc4781d7084900e6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/122433d501ce467f88d048808135f51f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/12ba21aafd1c4b8b80ba98ca0ebd0508, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/1a931aed88d746a7a0334ef576c3b2dd, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/dee770618b204057a19ad0f4969401da, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7686ce331ae740c5b83751f37275a5b7, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/6cfdc6cda95c47a3919ef4f9d53e8371, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/b79e2c00b20e4e83be7d2844716ee003] to archive 2023-04-01 14:56:30,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:56:30,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5ccdc27eb0704b908d36e610535e8885 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5ccdc27eb0704b908d36e610535e8885 2023-04-01 14:56:30,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d7c8e4d1229d4695a23bacf095693038 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d7c8e4d1229d4695a23bacf095693038 2023-04-01 14:56:30,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e9bd859fff7b4ac99ec01c86d28c48bf to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e9bd859fff7b4ac99ec01c86d28c48bf 2023-04-01 14:56:30,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d21d9f71e8a94901841f9416295defeb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d21d9f71e8a94901841f9416295defeb 2023-04-01 14:56:30,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/f0573b5473d04bdfb5de3dfe355f6c50 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/f0573b5473d04bdfb5de3dfe355f6c50 2023-04-01 14:56:30,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/03b514f2737a4b4b94131ea0102085da to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/03b514f2737a4b4b94131ea0102085da 2023-04-01 14:56:30,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/48aefbd1311b44e5ac98d956e27d7453 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/48aefbd1311b44e5ac98d956e27d7453 2023-04-01 14:56:30,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/c9d6012403b34eafae22a867a7584630 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/c9d6012403b34eafae22a867a7584630 2023-04-01 14:56:30,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5f6a85c4651246769a6c9207bf565714 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5f6a85c4651246769a6c9207bf565714 2023-04-01 14:56:30,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/19385fa9f688465a9cc48493e2b5c0b7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/19385fa9f688465a9cc48493e2b5c0b7 2023-04-01 14:56:30,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e67e8805beb8478ea198391d847540f6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e67e8805beb8478ea198391d847540f6 2023-04-01 14:56:30,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/cc33c609b12d456bb956de4c6d3c7579 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/cc33c609b12d456bb956de4c6d3c7579 2023-04-01 14:56:30,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/a5f33c5ef15746d381f5d99cb7bdfd76 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/a5f33c5ef15746d381f5d99cb7bdfd76 2023-04-01 14:56:30,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/40e7339460f1480a8d7848eccc5e3d3e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/40e7339460f1480a8d7848eccc5e3d3e 2023-04-01 14:56:30,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/3c86eb3446954587a43b93259222bcc7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/3c86eb3446954587a43b93259222bcc7 2023-04-01 14:56:30,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e914c686bf924e4c8509a8c7a2e8df31 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e914c686bf924e4c8509a8c7a2e8df31 2023-04-01 14:56:30,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/915d3fa7b4cf422195db4a7a2c0f5549 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/915d3fa7b4cf422195db4a7a2c0f5549 2023-04-01 14:56:30,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d190107168fb42f49dbb17b8bc6adfc9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/d190107168fb42f49dbb17b8bc6adfc9 2023-04-01 14:56:30,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/88fb8e21311a40a3a1e760cff5901908 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/88fb8e21311a40a3a1e760cff5901908 2023-04-01 14:56:30,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5b43a0a32d594a3ab0eac01fa48849da to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/5b43a0a32d594a3ab0eac01fa48849da 2023-04-01 14:56:30,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/38aaab2f28ea41369ce7ad0c58848591 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/38aaab2f28ea41369ce7ad0c58848591 2023-04-01 14:56:30,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/ae917c1cb389464a90af32d762073e02 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/ae917c1cb389464a90af32d762073e02 2023-04-01 14:56:30,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7b60d84e122d47babc4781d7084900e6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7b60d84e122d47babc4781d7084900e6 2023-04-01 14:56:30,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/122433d501ce467f88d048808135f51f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/122433d501ce467f88d048808135f51f 2023-04-01 14:56:30,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/12ba21aafd1c4b8b80ba98ca0ebd0508 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/12ba21aafd1c4b8b80ba98ca0ebd0508 2023-04-01 14:56:30,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/1a931aed88d746a7a0334ef576c3b2dd to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/1a931aed88d746a7a0334ef576c3b2dd 2023-04-01 14:56:30,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/dee770618b204057a19ad0f4969401da to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/dee770618b204057a19ad0f4969401da 2023-04-01 14:56:30,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7686ce331ae740c5b83751f37275a5b7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/7686ce331ae740c5b83751f37275a5b7 2023-04-01 14:56:30,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/6cfdc6cda95c47a3919ef4f9d53e8371 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/6cfdc6cda95c47a3919ef4f9d53e8371 2023-04-01 14:56:30,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/b79e2c00b20e4e83be7d2844716ee003 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/b79e2c00b20e4e83be7d2844716ee003 2023-04-01 14:56:30,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/da1f89fbc94b4280877a99ef5e210ea3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d87eb911a1644f2abadcd9f58247578a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/7ff83ee95a60400db3515daf128fc8c0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/45db72dc0c814017b5756292cf0bed9d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e5b31082b4244f039135cd376ec9ee0f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e63839d4f7c34628811f81430dec3ccd, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/53ff2aed5cf04a5f8aed209f727e9d96, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3ffd866b88fc4eed8e55058ff2742310, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/75f7d379aa9c4e559c17b811aacaefa6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d1869b4e375347108dd3fab26ee14627, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/6a825289214440f2b0f5157989f4226d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1281e45408f14f35ab812edd28eab5c0, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/5f8a06e578b541929e8b4d83024d1927, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cfc4ee8a684644898646eecb69899f92, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/eb93f1dd1a1040eba392d0e6c8d3021c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/85c7f4aae3bc4d71bdb8c570ab19cb2b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cc4d6faa774941eba193c508949d055b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cf7739d148014321931607a3899babcc, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/798382e17d014b71a79f0c56b8e8d4fa, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/8aeae02b878f43ccb7172a3f69fe9166, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e522c99371c84d7288463e3d6381e2c9, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3aae5ceffbaf4c248202bdde9670470c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/58904f78b40c4dce9dbe0c6527711c8e, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/f6aff69c537947ceb37901d7d9ed1c5c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d05248fd7ba740de94bd43e47eda8728, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/310fca4baa834bfbb5575fe6dba29b7c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2d50aa7c94094073a30f00e194edcc5f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1cc120fbce0944ab9b8678d171613cbb, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2299b55e9e6948d79a8a4f803f61932c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/6b1b5e4221134d7f9eab0b9cbdf7cef3] to archive 2023-04-01 14:56:30,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:56:30,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/da1f89fbc94b4280877a99ef5e210ea3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/da1f89fbc94b4280877a99ef5e210ea3 2023-04-01 14:56:30,623 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d87eb911a1644f2abadcd9f58247578a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d87eb911a1644f2abadcd9f58247578a 2023-04-01 14:56:30,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/7ff83ee95a60400db3515daf128fc8c0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/7ff83ee95a60400db3515daf128fc8c0 2023-04-01 14:56:30,625 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/45db72dc0c814017b5756292cf0bed9d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/45db72dc0c814017b5756292cf0bed9d 2023-04-01 14:56:30,626 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e5b31082b4244f039135cd376ec9ee0f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e5b31082b4244f039135cd376ec9ee0f 2023-04-01 14:56:30,626 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e63839d4f7c34628811f81430dec3ccd to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e63839d4f7c34628811f81430dec3ccd 2023-04-01 14:56:30,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/53ff2aed5cf04a5f8aed209f727e9d96 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/53ff2aed5cf04a5f8aed209f727e9d96 2023-04-01 14:56:30,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3ffd866b88fc4eed8e55058ff2742310 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3ffd866b88fc4eed8e55058ff2742310 2023-04-01 14:56:30,629 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/75f7d379aa9c4e559c17b811aacaefa6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/75f7d379aa9c4e559c17b811aacaefa6 2023-04-01 14:56:30,630 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d1869b4e375347108dd3fab26ee14627 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d1869b4e375347108dd3fab26ee14627 2023-04-01 14:56:30,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/6a825289214440f2b0f5157989f4226d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/6a825289214440f2b0f5157989f4226d 2023-04-01 14:56:30,632 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1281e45408f14f35ab812edd28eab5c0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1281e45408f14f35ab812edd28eab5c0 2023-04-01 14:56:30,632 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/5f8a06e578b541929e8b4d83024d1927 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/5f8a06e578b541929e8b4d83024d1927 2023-04-01 14:56:30,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cfc4ee8a684644898646eecb69899f92 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cfc4ee8a684644898646eecb69899f92 2023-04-01 14:56:30,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/eb93f1dd1a1040eba392d0e6c8d3021c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/eb93f1dd1a1040eba392d0e6c8d3021c 2023-04-01 14:56:30,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/85c7f4aae3bc4d71bdb8c570ab19cb2b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/85c7f4aae3bc4d71bdb8c570ab19cb2b 2023-04-01 14:56:30,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cc4d6faa774941eba193c508949d055b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cc4d6faa774941eba193c508949d055b 2023-04-01 14:56:30,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cf7739d148014321931607a3899babcc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/cf7739d148014321931607a3899babcc 2023-04-01 14:56:30,637 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/798382e17d014b71a79f0c56b8e8d4fa to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/798382e17d014b71a79f0c56b8e8d4fa 2023-04-01 14:56:30,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/8aeae02b878f43ccb7172a3f69fe9166 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/8aeae02b878f43ccb7172a3f69fe9166 2023-04-01 14:56:30,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e522c99371c84d7288463e3d6381e2c9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/e522c99371c84d7288463e3d6381e2c9 2023-04-01 14:56:30,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3aae5ceffbaf4c248202bdde9670470c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/3aae5ceffbaf4c248202bdde9670470c 2023-04-01 14:56:30,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/58904f78b40c4dce9dbe0c6527711c8e to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/58904f78b40c4dce9dbe0c6527711c8e 2023-04-01 14:56:30,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/f6aff69c537947ceb37901d7d9ed1c5c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/f6aff69c537947ceb37901d7d9ed1c5c 2023-04-01 14:56:30,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d05248fd7ba740de94bd43e47eda8728 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/d05248fd7ba740de94bd43e47eda8728 2023-04-01 14:56:30,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/310fca4baa834bfbb5575fe6dba29b7c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/310fca4baa834bfbb5575fe6dba29b7c 2023-04-01 14:56:30,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2d50aa7c94094073a30f00e194edcc5f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2d50aa7c94094073a30f00e194edcc5f 2023-04-01 14:56:30,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1cc120fbce0944ab9b8678d171613cbb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1cc120fbce0944ab9b8678d171613cbb 2023-04-01 14:56:30,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2299b55e9e6948d79a8a4f803f61932c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/2299b55e9e6948d79a8a4f803f61932c 2023-04-01 14:56:30,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/6b1b5e4221134d7f9eab0b9cbdf7cef3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/6b1b5e4221134d7f9eab0b9cbdf7cef3 2023-04-01 14:56:30,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44bcad13ffe24e46bd7cc6e5f71aaa0b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/db8586053c2f4a9199f70d9b9e4dda7d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f835066826c542d9a318df89fd567b6c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/938ddb9e82c64e9ea36f78bd90d4d7ca, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/22a76f3b5ffe482086f230ef6e09ed7d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/170c3dbb93e541c89380d062ddb9c1de, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/b13faa0600de460b84b2066a917fe7bd, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f5a76459ce264494b2d73340e3c5f8a6, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/ad792a1a106c4ed6ba3992e3183f76ca, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2872b4bd9e20456790c14bf7d85f0fad, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/de9345c40c8f43b2b060b9b01f9a1c7b, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/89a680c2209a46d7adf3ffac36ac6653, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44870f955c9f47dd91dfb27f186cbae2, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f83b144fbd854068b9e672169eb54e4c, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/10e3e3dafdd8452e9e23c721e66d4d31, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/cbb6f39d1368496fbd05f085ae63d336, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/bb84e8fb92f34b27851b22b8d3958535, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/0e4f8d834d6b4989adf50d69dc1b7b1d, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/e3d3f38c512e4da089d3f935a98dc349, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/df160d1ad4d648eaad59e3d623e4092a, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/ff1bb16f54b04e73b927ccd9b724ca2f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/106702735ed24477b4eced4153640f4f, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/3872819f04a54aecb82e37a5928d4659, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/a634f1afffb142cf8ac5607b30fa43a3, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/81df411ac7c94ccaa6a53486e02fed68, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/c3758a3e935642e29f8a2f5304921fe5, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/653293c34ea24e9f83a10de42fe3a519, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f9bd86836a534b1bb7f81a7a57ddca01, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/cd55fe3203d948038577842cb4121efe, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2af6349daab54f0fa2b9c2868de4a3b6] to archive 2023-04-01 14:56:30,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-01 14:56:30,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44bcad13ffe24e46bd7cc6e5f71aaa0b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44bcad13ffe24e46bd7cc6e5f71aaa0b 2023-04-01 14:56:30,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/db8586053c2f4a9199f70d9b9e4dda7d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/db8586053c2f4a9199f70d9b9e4dda7d 2023-04-01 14:56:30,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f835066826c542d9a318df89fd567b6c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f835066826c542d9a318df89fd567b6c 2023-04-01 14:56:30,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/938ddb9e82c64e9ea36f78bd90d4d7ca to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/938ddb9e82c64e9ea36f78bd90d4d7ca 2023-04-01 14:56:30,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/22a76f3b5ffe482086f230ef6e09ed7d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/22a76f3b5ffe482086f230ef6e09ed7d 2023-04-01 14:56:30,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/170c3dbb93e541c89380d062ddb9c1de to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/170c3dbb93e541c89380d062ddb9c1de 2023-04-01 14:56:30,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/b13faa0600de460b84b2066a917fe7bd to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/b13faa0600de460b84b2066a917fe7bd 2023-04-01 14:56:30,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f5a76459ce264494b2d73340e3c5f8a6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f5a76459ce264494b2d73340e3c5f8a6 2023-04-01 14:56:30,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/ad792a1a106c4ed6ba3992e3183f76ca to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/ad792a1a106c4ed6ba3992e3183f76ca 2023-04-01 14:56:30,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2872b4bd9e20456790c14bf7d85f0fad to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2872b4bd9e20456790c14bf7d85f0fad 2023-04-01 14:56:30,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/de9345c40c8f43b2b060b9b01f9a1c7b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/de9345c40c8f43b2b060b9b01f9a1c7b 2023-04-01 14:56:30,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/89a680c2209a46d7adf3ffac36ac6653 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/89a680c2209a46d7adf3ffac36ac6653 2023-04-01 14:56:30,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44870f955c9f47dd91dfb27f186cbae2 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/44870f955c9f47dd91dfb27f186cbae2 2023-04-01 14:56:30,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f83b144fbd854068b9e672169eb54e4c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f83b144fbd854068b9e672169eb54e4c 2023-04-01 14:56:30,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/10e3e3dafdd8452e9e23c721e66d4d31 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/10e3e3dafdd8452e9e23c721e66d4d31 2023-04-01 14:56:30,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/cbb6f39d1368496fbd05f085ae63d336 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/cbb6f39d1368496fbd05f085ae63d336 2023-04-01 14:56:30,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/bb84e8fb92f34b27851b22b8d3958535 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/bb84e8fb92f34b27851b22b8d3958535 2023-04-01 14:56:30,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/0e4f8d834d6b4989adf50d69dc1b7b1d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/0e4f8d834d6b4989adf50d69dc1b7b1d 2023-04-01 14:56:30,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/e3d3f38c512e4da089d3f935a98dc349 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/e3d3f38c512e4da089d3f935a98dc349 2023-04-01 14:56:30,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/df160d1ad4d648eaad59e3d623e4092a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/df160d1ad4d648eaad59e3d623e4092a 2023-04-01 14:56:30,669 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/ff1bb16f54b04e73b927ccd9b724ca2f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/ff1bb16f54b04e73b927ccd9b724ca2f 2023-04-01 14:56:30,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/106702735ed24477b4eced4153640f4f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/106702735ed24477b4eced4153640f4f 2023-04-01 14:56:30,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/3872819f04a54aecb82e37a5928d4659 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/3872819f04a54aecb82e37a5928d4659 2023-04-01 14:56:30,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/a634f1afffb142cf8ac5607b30fa43a3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/a634f1afffb142cf8ac5607b30fa43a3 2023-04-01 14:56:30,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/81df411ac7c94ccaa6a53486e02fed68 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/81df411ac7c94ccaa6a53486e02fed68 2023-04-01 14:56:30,673 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/c3758a3e935642e29f8a2f5304921fe5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/c3758a3e935642e29f8a2f5304921fe5 2023-04-01 14:56:30,673 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/653293c34ea24e9f83a10de42fe3a519 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/653293c34ea24e9f83a10de42fe3a519 2023-04-01 14:56:30,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f9bd86836a534b1bb7f81a7a57ddca01 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/f9bd86836a534b1bb7f81a7a57ddca01 2023-04-01 14:56:30,675 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/cd55fe3203d948038577842cb4121efe to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/cd55fe3203d948038577842cb4121efe 2023-04-01 14:56:30,676 DEBUG [StoreCloser-TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2af6349daab54f0fa2b9c2868de4a3b6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/2af6349daab54f0fa2b9c2868de4a3b6 2023-04-01 14:56:30,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/recovered.edits/1496.seqid, newMaxSeqId=1496, maxSeqId=4 2023-04-01 14:56:30,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4. 2023-04-01 14:56:30,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d5537299121eaceb1e94c26142bdabf4: 2023-04-01 14:56:30,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:30,682 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=d5537299121eaceb1e94c26142bdabf4, regionState=CLOSED 2023-04-01 14:56:30,682 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680360990682"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680360990682"}]},"ts":"1680360990682"} 2023-04-01 14:56:30,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-01 14:56:30,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure d5537299121eaceb1e94c26142bdabf4, server=jenkins-hbase4.apache.org,40863,1680360818567 in 1.5970 sec 2023-04-01 14:56:30,685 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-01 14:56:30,685 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d5537299121eaceb1e94c26142bdabf4, UNASSIGN in 1.6010 sec 2023-04-01 14:56:30,685 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680360990685"}]},"ts":"1680360990685"} 2023-04-01 14:56:30,686 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-01 14:56:30,687 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-01 14:56:30,688 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6100 sec 2023-04-01 14:56:31,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-01 14:56:31,184 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-01 14:56:31,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-01 14:56:31,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:31,187 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:31,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-01 14:56:31,188 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:31,192 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:31,194 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C, FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/recovered.edits] 2023-04-01 14:56:31,197 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/03d3af142da447058fcfb3e37fcbd668 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/03d3af142da447058fcfb3e37fcbd668 2023-04-01 14:56:31,198 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/274e55b3bf174533958b0ca0b18bb16f to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/274e55b3bf174533958b0ca0b18bb16f 2023-04-01 14:56:31,199 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/75b796e694ac4d5595ecfec5fdb18e0b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/75b796e694ac4d5595ecfec5fdb18e0b 2023-04-01 14:56:31,200 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/b59e2d636c974c08817d036d62e21cc9 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/b59e2d636c974c08817d036d62e21cc9 2023-04-01 14:56:31,201 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/df6dfcf5750e4c26962897e4326b8d1d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/df6dfcf5750e4c26962897e4326b8d1d 2023-04-01 14:56:31,202 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e872bde770004631a241621b641033cb to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/A/e872bde770004631a241621b641033cb 2023-04-01 14:56:31,205 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/0fcb1580f0d14d9cbcc558690bfe3758 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/0fcb1580f0d14d9cbcc558690bfe3758 2023-04-01 14:56:31,206 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1aa2b8dc9c8842ab8bc3b3e7c4f99352 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/1aa2b8dc9c8842ab8bc3b3e7c4f99352 2023-04-01 14:56:31,207 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/73f8a8b1b7df4a0fa40dc4ee3a6f3e5d to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/73f8a8b1b7df4a0fa40dc4ee3a6f3e5d 2023-04-01 14:56:31,208 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/923f283b6f1848ae9a40fcfe677296f5 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/923f283b6f1848ae9a40fcfe677296f5 2023-04-01 14:56:31,209 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/acf83307a17e456caf64f46ccb78de27 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/acf83307a17e456caf64f46ccb78de27 2023-04-01 14:56:31,210 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/b8d581add7cc4f4e8bca5934ec392c01 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/B/b8d581add7cc4f4e8bca5934ec392c01 2023-04-01 14:56:31,212 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/09ccf2544517411bb721f66c7400b627 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/09ccf2544517411bb721f66c7400b627 2023-04-01 14:56:31,213 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/4d3f05b7777344648cb434c33a73aaa3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/4d3f05b7777344648cb434c33a73aaa3 2023-04-01 14:56:31,214 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/5f457a6f911543b187a4a23ba4171752 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/5f457a6f911543b187a4a23ba4171752 2023-04-01 14:56:31,215 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/9058501263064e11869a21dc7cf8bd64 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/9058501263064e11869a21dc7cf8bd64 2023-04-01 14:56:31,216 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/b9a1f781bb7a4c53a83fd35c5462e7c7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/b9a1f781bb7a4c53a83fd35c5462e7c7 2023-04-01 14:56:31,217 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/fb1f381c17a04d6c88d857a30c16eec7 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/C/fb1f381c17a04d6c88d857a30c16eec7 2023-04-01 14:56:31,219 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/recovered.edits/1496.seqid to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4/recovered.edits/1496.seqid 2023-04-01 14:56:31,219 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/.tmp/data/default/TestAcidGuarantees/d5537299121eaceb1e94c26142bdabf4 2023-04-01 14:56:31,219 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-01 14:56:31,220 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 14:56:31,220 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-01 14:56:31,223 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011902e073df88467f9d81ae40efb3fe3c to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304011902e073df88467f9d81ae40efb3fe3c 2023-04-01 14:56:31,224 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040120f619f4bb39458fa48ec7e2d6e11c61 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040120f619f4bb39458fa48ec7e2d6e11c61 2023-04-01 14:56:31,225 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401223f0f5f9f0c4e85ac4585718c916e43 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401223f0f5f9f0c4e85ac4585718c916e43 2023-04-01 14:56:31,226 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012faaf7578a4741c3953b4282736b377a to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304012faaf7578a4741c3953b4282736b377a 2023-04-01 14:56:31,227 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040134a2b0caab93425a9558adfc7df893f0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040134a2b0caab93425a9558adfc7df893f0 2023-04-01 14:56:31,228 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013618684a8fc443ea88571048c1e08416 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013618684a8fc443ea88571048c1e08416 2023-04-01 14:56:31,229 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013dc675cb0e3c47bcbb2dfc88ae354143 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304013dc675cb0e3c47bcbb2dfc88ae354143 2023-04-01 14:56:31,230 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040148d430f0eb8647ac9d25288f0c6e1c88 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040148d430f0eb8647ac9d25288f0c6e1c88 2023-04-01 14:56:31,231 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040151bf9679df2e405593656865310b1414 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040151bf9679df2e405593656865310b1414 2023-04-01 14:56:31,232 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040155cf2adbd350469e98f273b7bdedd343 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040155cf2adbd350469e98f273b7bdedd343 2023-04-01 14:56:31,233 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015aa480817d894af496360ecdb2249805 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015aa480817d894af496360ecdb2249805 2023-04-01 14:56:31,234 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015b6414e89e1a4fdf8fa4ff9bfe3c2607 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015b6414e89e1a4fdf8fa4ff9bfe3c2607 2023-04-01 14:56:31,235 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015df83c7035c54a9cb4d8d47ea7a4f579 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304015df83c7035c54a9cb4d8d47ea7a4f579 2023-04-01 14:56:31,236 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401778298f508b14453bab5a7d76cb09b28 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401778298f508b14453bab5a7d76cb09b28 2023-04-01 14:56:31,237 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401788a434aa1b146828197810c79750f3b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401788a434aa1b146828197810c79750f3b 2023-04-01 14:56:31,239 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018bf3fd2783ba446784c28e8fb0ed4d3b to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304018bf3fd2783ba446784c28e8fb0ed4d3b 2023-04-01 14:56:31,240 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ae16207424cf4af4a654a679d751f677 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ae16207424cf4af4a654a679d751f677 2023-04-01 14:56:31,241 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401afaa8ca0a3a54a8f9079f786f91efcef to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401afaa8ca0a3a54a8f9079f786f91efcef 2023-04-01 14:56:31,242 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401afe12bbac3184868b74b60f3840640dc to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401afe12bbac3184868b74b60f3840640dc 2023-04-01 14:56:31,243 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b241d29496be4c7d9fa255a0bb212887 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b241d29496be4c7d9fa255a0bb212887 2023-04-01 14:56:31,245 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b6ca74388fb941d39bd2903f66efb926 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b6ca74388fb941d39bd2903f66efb926 2023-04-01 14:56:31,246 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b8018d3d909043e1a98bc5fcb110faf1 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401b8018d3d909043e1a98bc5fcb110faf1 2023-04-01 14:56:31,247 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c0a0e3a98aae4f809a0eea41c1b25f14 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401c0a0e3a98aae4f809a0eea41c1b25f14 2023-04-01 14:56:31,248 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401d27604c0b9b642c3b84ed083095f70e3 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401d27604c0b9b642c3b84ed083095f70e3 2023-04-01 14:56:31,250 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401d82e581e12a54b5d82d4c567b532bd86 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401d82e581e12a54b5d82d4c567b532bd86 2023-04-01 14:56:31,251 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ddafc908e959479b92adc260009778ef to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ddafc908e959479b92adc260009778ef 2023-04-01 14:56:31,252 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401df233e0004cb43379dde95e163596db6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401df233e0004cb43379dde95e163596db6 2023-04-01 14:56:31,253 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e843cc1df392489eb41a1905c07badc0 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401e843cc1df392489eb41a1905c07badc0 2023-04-01 14:56:31,254 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ed3290173e764308a1d7bd4460e7d6d6 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401ed3290173e764308a1d7bd4460e7d6d6 2023-04-01 14:56:31,255 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401fd0d66aa6c3446b1a78d27ef34c77cf4 to hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230401fd0d66aa6c3446b1a78d27ef34c77cf4 2023-04-01 14:56:31,255 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-01 14:56:31,257 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:31,259 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-01 14:56:31,260 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-01 14:56:31,261 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:31,261 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-01 14:56:31,261 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680360991261"}]},"ts":"9223372036854775807"} 2023-04-01 14:56:31,263 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-01 14:56:31,263 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d5537299121eaceb1e94c26142bdabf4, NAME => 'TestAcidGuarantees,,1680360965567.d5537299121eaceb1e94c26142bdabf4.', STARTKEY => '', ENDKEY => ''}] 2023-04-01 14:56:31,263 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-01 14:56:31,263 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680360991263"}]},"ts":"9223372036854775807"} 2023-04-01 14:56:31,264 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-01 14:56:31,265 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-01 14:56:31,266 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 80 msec 2023-04-01 14:56:31,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35215] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-01 14:56:31,289 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-01 14:56:31,303 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229 (was 234), OpenFileDescriptor=571 (was 618), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=420 (was 413) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=3496 (was 3301) - AvailableMemoryMB LEAK? - 2023-04-01 14:56:31,303 INFO [Listener at localhost/46667] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-01 14:56:31,303 INFO [Listener at localhost/46667] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-01 14:56:31,303 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ddd91d3 to 127.0.0.1:60465 2023-04-01 14:56:31,304 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:31,304 DEBUG [Listener at localhost/46667] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-01 14:56:31,304 DEBUG [Listener at localhost/46667] util.JVMClusterUtil(257): Found active master hash=954520778, stopped=false 2023-04-01 14:56:31,305 INFO [Listener at localhost/46667] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,35215,1680360817296 2023-04-01 14:56:31,306 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-01 14:56:31,306 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(629): Stopping 2023-04-01 14:56:31,306 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 14:56:31,306 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-01 14:56:31,306 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30d22915 to 127.0.0.1:60465 2023-04-01 14:56:31,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-01 14:56:31,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-01 14:56:31,306 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:31,307 INFO [Listener at localhost/46667] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40863,1680360818567' ***** 2023-04-01 14:56:31,307 INFO [Listener at localhost/46667] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-01 14:56:31,307 INFO [RS:0;jenkins-hbase4:40863] regionserver.HeapMemoryManager(220): Stopping 2023-04-01 14:56:31,307 INFO [RS:0;jenkins-hbase4:40863] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-01 14:56:31,307 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-01 14:56:31,307 INFO [RS:0;jenkins-hbase4:40863] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-01 14:56:31,308 INFO [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(3304): Received CLOSE for 258dfbc10657b5aa46b7285094b199c1 2023-04-01 14:56:31,308 INFO [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:31,308 DEBUG [RS:0;jenkins-hbase4:40863] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e83673 to 127.0.0.1:60465 2023-04-01 14:56:31,309 DEBUG [RS:0;jenkins-hbase4:40863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:31,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 258dfbc10657b5aa46b7285094b199c1, disabling compactions & flushes 2023-04-01 14:56:31,309 INFO [RS:0;jenkins-hbase4:40863] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-01 14:56:31,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:56:31,309 INFO [RS:0;jenkins-hbase4:40863] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-01 14:56:31,309 INFO [RS:0;jenkins-hbase4:40863] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-01 14:56:31,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:56:31,309 INFO [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-01 14:56:31,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. after waiting 0 ms 2023-04-01 14:56:31,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:56:31,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 258dfbc10657b5aa46b7285094b199c1 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-01 14:56:31,309 INFO [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-01 14:56:31,309 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 258dfbc10657b5aa46b7285094b199c1=hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1.} 2023-04-01 14:56:31,310 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-01 14:56:31,310 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-01 14:56:31,310 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-01 14:56:31,310 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-01 14:56:31,310 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-01 14:56:31,310 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-01 14:56:31,311 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1505): Waiting on 1588230740, 258dfbc10657b5aa46b7285094b199c1 2023-04-01 14:56:31,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1/.tmp/info/5df9078a516c402c8d7b0bb20a011ffb 2023-04-01 14:56:31,320 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1/.tmp/info/5df9078a516c402c8d7b0bb20a011ffb as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1/info/5df9078a516c402c8d7b0bb20a011ffb 2023-04-01 14:56:31,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1/info/5df9078a516c402c8d7b0bb20a011ffb, entries=2, sequenceid=6, filesize=4.8 K 2023-04-01 14:56:31,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 258dfbc10657b5aa46b7285094b199c1 in 15ms, sequenceid=6, compaction requested=false 2023-04-01 14:56:31,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/namespace/258dfbc10657b5aa46b7285094b199c1/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-01 14:56:31,328 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-01 14:56:31,329 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:56:31,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 258dfbc10657b5aa46b7285094b199c1: 2023-04-01 14:56:31,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680360821207.258dfbc10657b5aa46b7285094b199c1. 2023-04-01 14:56:31,512 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-01 14:56:31,712 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-01 14:56:31,718 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/.tmp/info/aef163793c774fe489e2363023fbc72d 2023-04-01 14:56:31,721 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for aef163793c774fe489e2363023fbc72d 2023-04-01 14:56:31,733 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/.tmp/rep_barrier/64daafb399eb4c15b9b7b6035ba2c2f8 2023-04-01 14:56:31,736 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 64daafb399eb4c15b9b7b6035ba2c2f8 2023-04-01 14:56:31,905 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-01 14:56:31,905 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-01 14:56:31,912 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-01 14:56:32,113 DEBUG [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-01 14:56:32,141 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/.tmp/table/6ea45689ee25409da31f8e4145820f5c 2023-04-01 14:56:32,144 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 6ea45689ee25409da31f8e4145820f5c 2023-04-01 14:56:32,144 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/.tmp/info/aef163793c774fe489e2363023fbc72d as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/info/aef163793c774fe489e2363023fbc72d 2023-04-01 14:56:32,147 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for aef163793c774fe489e2363023fbc72d 2023-04-01 14:56:32,147 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/info/aef163793c774fe489e2363023fbc72d, entries=22, sequenceid=93, filesize=7.1 K 2023-04-01 14:56:32,148 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/.tmp/rep_barrier/64daafb399eb4c15b9b7b6035ba2c2f8 as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/rep_barrier/64daafb399eb4c15b9b7b6035ba2c2f8 2023-04-01 14:56:32,151 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 64daafb399eb4c15b9b7b6035ba2c2f8 2023-04-01 14:56:32,151 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/rep_barrier/64daafb399eb4c15b9b7b6035ba2c2f8, entries=6, sequenceid=93, filesize=5.5 K 2023-04-01 14:56:32,151 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/.tmp/table/6ea45689ee25409da31f8e4145820f5c as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/table/6ea45689ee25409da31f8e4145820f5c 2023-04-01 14:56:32,154 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 6ea45689ee25409da31f8e4145820f5c 2023-04-01 14:56:32,154 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/table/6ea45689ee25409da31f8e4145820f5c, entries=9, sequenceid=93, filesize=5.5 K 2023-04-01 14:56:32,154 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 844ms, sequenceid=93, compaction requested=false 2023-04-01 14:56:32,158 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-01 14:56:32,158 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-01 14:56:32,159 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-01 14:56:32,159 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-01 14:56:32,160 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-01 14:56:32,313 INFO [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40863,1680360818567; all regions closed. 2023-04-01 14:56:32,321 DEBUG [RS:0;jenkins-hbase4:40863] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/oldWALs 2023-04-01 14:56:32,321 INFO [RS:0;jenkins-hbase4:40863] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40863%2C1680360818567.meta:.meta(num 1680360820954) 2023-04-01 14:56:32,326 DEBUG [RS:0;jenkins-hbase4:40863] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/oldWALs 2023-04-01 14:56:32,326 INFO [RS:0;jenkins-hbase4:40863] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40863%2C1680360818567:(num 1680360820060) 2023-04-01 14:56:32,326 DEBUG [RS:0;jenkins-hbase4:40863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:32,326 INFO [RS:0;jenkins-hbase4:40863] regionserver.LeaseManager(133): Closed leases 2023-04-01 14:56:32,326 INFO [RS:0;jenkins-hbase4:40863] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-01 14:56:32,326 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-01 14:56:32,328 INFO [RS:0;jenkins-hbase4:40863] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40863 2023-04-01 14:56:32,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40863,1680360818567 2023-04-01 14:56:32,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-01 14:56:32,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-01 14:56:32,337 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6a355721 rejected from java.util.concurrent.ThreadPoolExecutor@6e0adebf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2306] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-01 14:56:32,341 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,40863,1680360818567] 2023-04-01 14:56:32,341 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40863,1680360818567; numProcessing=1 2023-04-01 14:56:32,343 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40863,1680360818567 already deleted, retry=false 2023-04-01 14:56:32,343 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,40863,1680360818567 expired; onlineServers=0 2023-04-01 14:56:32,343 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35215,1680360817296' ***** 2023-04-01 14:56:32,343 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-01 14:56:32,343 DEBUG [M:0;jenkins-hbase4:35215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79d9def1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-01 14:56:32,343 INFO [M:0;jenkins-hbase4:35215] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35215,1680360817296 2023-04-01 14:56:32,343 INFO [M:0;jenkins-hbase4:35215] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35215,1680360817296; all regions closed. 2023-04-01 14:56:32,343 DEBUG [M:0;jenkins-hbase4:35215] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-01 14:56:32,343 DEBUG [M:0;jenkins-hbase4:35215] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-01 14:56:32,343 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-01 14:56:32,343 DEBUG [M:0;jenkins-hbase4:35215] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-01 14:56:32,343 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680360819688] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680360819688,5,FailOnTimeoutGroup] 2023-04-01 14:56:32,343 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680360819683] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680360819683,5,FailOnTimeoutGroup] 2023-04-01 14:56:32,343 INFO [M:0;jenkins-hbase4:35215] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-01 14:56:32,343 INFO [M:0;jenkins-hbase4:35215] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-01 14:56:32,344 INFO [M:0;jenkins-hbase4:35215] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-01 14:56:32,344 DEBUG [M:0;jenkins-hbase4:35215] master.HMaster(1512): Stopping service threads 2023-04-01 14:56:32,344 INFO [M:0;jenkins-hbase4:35215] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-01 14:56:32,344 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-01 14:56:32,344 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-01 14:56:32,345 ERROR [M:0;jenkins-hbase4:35215] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (1612801661) connection to localhost/127.0.0.1:39359 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-01 14:56:32,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-01 14:56:32,345 INFO [M:0;jenkins-hbase4:35215] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-01 14:56:32,345 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-01 14:56:32,345 DEBUG [M:0;jenkins-hbase4:35215] zookeeper.ZKUtil(398): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-01 14:56:32,346 WARN [M:0;jenkins-hbase4:35215] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-01 14:56:32,346 INFO [M:0;jenkins-hbase4:35215] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-01 14:56:32,346 INFO [M:0;jenkins-hbase4:35215] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-01 14:56:32,346 DEBUG [M:0;jenkins-hbase4:35215] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-01 14:56:32,346 INFO [M:0;jenkins-hbase4:35215] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 14:56:32,346 DEBUG [M:0;jenkins-hbase4:35215] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 14:56:32,346 DEBUG [M:0;jenkins-hbase4:35215] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-01 14:56:32,346 DEBUG [M:0;jenkins-hbase4:35215] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 14:56:32,346 INFO [M:0;jenkins-hbase4:35215] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.06 KB heapSize=393.77 KB 2023-04-01 14:56:32,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-01 14:56:32,442 INFO [RS:0;jenkins-hbase4:40863] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40863,1680360818567; zookeeper connection closed. 2023-04-01 14:56:32,442 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5a25c979 rejected from java.util.concurrent.ThreadPoolExecutor@6e0adebf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2306] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-01 14:56:32,442 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:40863-0x101403728ea0001, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-01 14:56:32,442 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@c75dbfc] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@c75dbfc 2023-04-01 14:56:32,442 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@22aedb97 rejected from java.util.concurrent.ThreadPoolExecutor@6e0adebf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2306] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-01 14:56:32,442 INFO [Listener at localhost/46667] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-01 14:56:32,758 INFO [M:0;jenkins-hbase4:35215] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.06 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/60459ba70f884e659193a2bd5677236b 2023-04-01 14:56:32,761 INFO [M:0;jenkins-hbase4:35215] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 60459ba70f884e659193a2bd5677236b 2023-04-01 14:56:32,761 DEBUG [M:0;jenkins-hbase4:35215] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/60459ba70f884e659193a2bd5677236b as hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/60459ba70f884e659193a2bd5677236b 2023-04-01 14:56:32,764 INFO [M:0;jenkins-hbase4:35215] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 60459ba70f884e659193a2bd5677236b 2023-04-01 14:56:32,764 INFO [M:0;jenkins-hbase4:35215] regionserver.HStore(1080): Added hdfs://localhost:39359/user/jenkins/test-data/01c2ecad-0db1-b90a-edea-d5eb4ff1ab2a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/60459ba70f884e659193a2bd5677236b, entries=65, sequenceid=578, filesize=24.7 K 2023-04-01 14:56:32,765 INFO [M:0;jenkins-hbase4:35215] regionserver.HRegion(2948): Finished flush of dataSize ~345.06 KB/353345, heapSize ~393.75 KB/403200, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 419ms, sequenceid=578, compaction requested=false 2023-04-01 14:56:32,765 INFO [M:0;jenkins-hbase4:35215] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-01 14:56:32,766 DEBUG [M:0;jenkins-hbase4:35215] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-01 14:56:32,768 INFO [M:0;jenkins-hbase4:35215] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-01 14:56:32,768 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-01 14:56:32,769 INFO [M:0;jenkins-hbase4:35215] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35215 2023-04-01 14:56:32,772 DEBUG [M:0;jenkins-hbase4:35215] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,35215,1680360817296 already deleted, retry=false 2023-04-01 14:56:32,873 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-01 14:56:32,873 INFO [M:0;jenkins-hbase4:35215] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35215,1680360817296; zookeeper connection closed. 2023-04-01 14:56:32,873 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@41fd6ecc rejected from java.util.concurrent.ThreadPoolExecutor@5d130203[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3854] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-01 14:56:32,873 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:35215-0x101403728ea0000, quorum=127.0.0.1:60465, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-01 14:56:32,873 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2970a6e5 rejected from java.util.concurrent.ThreadPoolExecutor@5d130203[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3854] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-01 14:56:32,875 WARN [Listener at localhost/46667] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-01 14:56:32,879 INFO [Listener at localhost/46667] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-01 14:56:32,986 WARN [BP-34758962-172.31.14.131-1680360813934 heartbeating to localhost/127.0.0.1:39359] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-01 14:56:32,987 WARN [BP-34758962-172.31.14.131-1680360813934 heartbeating to localhost/127.0.0.1:39359] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-34758962-172.31.14.131-1680360813934 (Datanode Uuid 350c837c-6d4c-42a4-a453-bc0081ccae8d) service to localhost/127.0.0.1:39359 2023-04-01 14:56:32,989 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/cluster_9b9f408e-8fe1-e5f9-3143-13e52c404ab9/dfs/data/data1/current/BP-34758962-172.31.14.131-1680360813934] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-01 14:56:32,989 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f27cdc20-2258-5871-3409-ea7ee303be71/cluster_9b9f408e-8fe1-e5f9-3143-13e52c404ab9/dfs/data/data2/current/BP-34758962-172.31.14.131-1680360813934] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-01 14:56:33,031 INFO [Listener at localhost/46667] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-01 14:56:33,201 INFO [Listener at localhost/46667] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-01 14:56:33,392 INFO [Listener at localhost/46667] hbase.HBaseTestingUtility(1293): Minicluster is down